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Abstract 

The purpose of this project was to design, produce, and field-test two computer-based 
interactive CD-ROMs: PEGS! for PRESCHOOL and PEGS! for SECONDARY. These 
programs, in a game format, provide beginning general and special educators with 
independent practice in using basic behavior management strategies. The 43 teachers and 
assistant teachers who volunteered to evaluate the PEGS! CD-ROM programs gave high 
ratings of approval overall and on each quality criterion: relevant content, a helpful 
learning experience, and easy to use. 

A pre- to post-CDR follow-up study in 23 classrooms showed statistically significant 
increases in the skills of participants after using the programs independently during a 2- 
week period. Average skills in behavior management increased 41% for 12 early 
childhood educators and 42% for 1 1 middle and high school teachers. Mean gains were 
statistically significant for both groups (p < .01 for early childhood; p < .04 for secondary 
teachers). Additionally, negative responses from children decreased by 64% for early 
childhood educators and by 28% for secondary school teachers. 

While the sample sizes for these preliminary field tests were small, the results hold 
promise that the PEGS! programs can be effective inservice training tools for 
independent adult learning and enhanced behavior management skills. 
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Executive Brief 

The project purpose was to design, produce, and field-test two computer-based interactive 
CD-ROMs: PEGS! for PRESCHOOL and PEGS! for SECONDARY. These programs, in 
game format, provide beginning general and special educators with independent practice 
using basic behavior management strategies in simulations of typical school-related 
activities. 



Each version contains animations of children behaving in familiar ways. The menus provide 
options for obtaining in-depth text information about each child character, the challenging 
behaviors they exhibit, and the intervention strategies. There are also three levels of 
difficulty or a practice session for warm-up. 

As an activity begins, the player selects a particular child and then chooses among 12 basic 
positive management strategies with the goal of increasing or maintaining the child's 
participation in the activity. When the choice is a good match for that individual child, there 
will be an increase in participation and a decrease in problem behavior. When a choice is not 
a good match, problem behavior increases. While a player attends to one child, the other 
children continue to engage in a range of behaviors. If ignored for too long, the others may 
exhibit negative, non-participatory behavior, and the game is lost. Alternative responses can 
be explored by repeating the same activity but choosing different strategies. Each time the 
game is played, a randomly selected small group of children— each with a different 
configuration of behavior problems— is presented. The game is won when all of the children 
are participating with enthusiasm, and full participation by the group is maintained. At the 
end, the program provides feedback about each child's participation and an analysis of the 
management strategies used. This feedback has a print option. 

The 43 teachers and assistant teachers who volunteered to evaluate the PEGS! CD-ROM 
programs gave high ratings of approval overall and on each quality criterion: relevant 
content, a helpful learning experience, and easy to use. 

A pre- to post-CDR follow-up study in 23 classrooms showed statistically significant 
increases in the skills of participants after using the program independently during a 2-week 
period. Average skills in behavior management increased 41% for 12 early childhood 
educators and 42% for 1 1 middle and high school teachers. Mean gains were statistically 
significant for both groups ( t = 2.53, p < .01 for early childhood; t = 1.91, p < .04 for 
secondary teachers). Additionally, negative responses from children decreased by 64% for 
early childhood educators and by 28% for the middle and high school teachers. 
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Among the early childhood educators, there was also a 30% decrease in frequency of 
interventions used, and 90% reduction in the frequency of participants' ignoring. Although 
participants who were less skilled and less experienced had the greatest benefit, experienced 
and skilled educators also made gains in skills and reduced the frequency of their 
interventions. 
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In contrast, for middle and high school teachers, who also made statistically significant 
improvement in their overall skill scores, there was little change in the frequency of their 
interventions and instances of ignoring students. However, there was a statistically 
significant difference between female and male teachers in the mean number of negative 
student responses before using the CDR. Female teachers had a total of 5 negative responses 
from students, while male teachers had a total of 83 instances of negative responses. For 
post-CDR observations, the female teachers had a total of two negative responses while the 
male teachers had 6 1 negative student responses (a 26% decrease). 

While the sample sizes for these preliminary field tests were small, the results hold promise 
that the PEGS! programs can be effective, inservice training tools for independent adult 
learning and enhanced classroom skills. 
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FINAL PERFORMANCE REPORT 

Project PEGS! Practice in Effective Guidance 

Strategies 

(CFDANo. 84.325 N) 
October 1, 1999 — September 30, 2002 
(no cost extension to December 30, 2002) 



INTRODUCTION 

The mission of Project PEGS! Practice in Effective Guidance Strategies (CFDA No. 84.325 N) was to 
design and produce two interactive computer-based CD-ROM (CDR) simulated programs for adult 
learning in game formats: (1) PEGS! for PRESCHOOL and (2) PEGS! for SECONDARY. The target users 
were beginning educators in general and special education — teachers and assistant teachers and others 
seeking to improve their classroom behavior management skills. The preschool version was targeted for 
those teaching children ages 2-5 years; and the secondary version, for those teaching students ages 12-16 
in middle or high school. 1 

The need. The project was in response to the 
expressed need among both general and special educators 
for skill training to use positive, developmentally 
appropriate behavior management strategies proven 
effective in natural settings with all children — with or 
without disabilities. The inclusion policies advocated by 
IDEA legislation; the Council for Exceptional Children; Council for Children with Behavioral 
Disabilities, Division for Early Childhood; and the National Association for the Education of Young 

1 An earlier version, funded by the Georgia Department of Education State Improvement Grant (SIG) in 1998-99, 
was the original prototype — a version developed for educators teaching elementary school age children. 
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Children have intensified the necessity for training both groups of educators of children who have social, 
emotional, or behavioral disabilities. Yet hurdles to meaningful training in effective behavior 
management have made training difficult to accomplish. These realities include finding time for training, 
the need for applied practice and feedback, diverse levels of skill mastery, individual differences in 
teachers’ background preparation, different rates of skill acquisition, and variations in prior exposure to a 
range of children’s behavior problems. Many schools also have a significant attrition of qualified teachers 
and staff which results in a need for repeated training in the basics. Periodic up-dating of skills is often 
necessary. Reliance on substitute teachers, out-of-field assignments, and paraprofessional turnover further 
complicate scheduling and implementing staff training programs. 

PEGS! CDRs address these obstacles to training. There is little prerequisite information needed to 
use them, and they require no group instructor and no meetings, thereby avoiding numerous problems 
associated with group training. They also offer guided skill practice, repeated practice, individualized 
feedback, and supplementary information about the strategies and children's behavior problems. This is a 
highly useful approach to adult learning — a cost-effective and time-efficient way to provide opportunities 
for skill building in using basic behavior management strategies successfully. 

The standards. PEGS! products were expected to achieve standards of relevant content, helpful 
independent learning experience, and ease of use as judged by the potential user groups. Table 1 lists 
these standards. The products were also expected to have a demonstrable carryover effect in actual 
classroom practices. Four outcome dimensions were established to reflect positive carryover into 
classroom application: (a) Educators' skill in using the selected strategies should increase, while there 
should be decreases in (b) the frequency with which they use intervention strategies, (c) students' negative 
responses to the strategies, and (d) ignoring of students' behavior. This report summarizes the steps taken 
to assure that the products met these standards. It also presents results of the formative and summative 
evaluation field tests. 



O 

ERLC 



9 



Project PEGS! CD-ROM, Page 3 



Table L Criteria^ Used, to Rate and. Evaluate, the PEGS . 7 Products. 

About the Content 



The PEGS! child characters have behaviors seen in real children. 

Their difficult behaviors are real-life challenges. 

The activities are typical of educational settings. 

The strategies are known to most educators. 

The child characters respond to the interventions in real-life ways. 

The responses of the characters are appropriate for their ages and individual needs. 

About the CDRs as Learning Opportunities 



The simulations hold users’ attention. 

Some behaviors are easy to manage. 

Other behaviors are difficult to manage. 

The program choices allow users to select their preferences. 

There is sufficient information about the characters to match strategies to their needs. 
Information about the characters is presented in a professional way. 

The strategies are practical for real-life situations. 

Feedback at the end provides useful information. 

About Using the CDRs with Ease 

They can be used with very little computer experience. 

Operating the program presents no problems. 

The format is easy to follow. 

Information about how to play is clear. 

It is easy to change from one activity to another in the program. 

Stopping and reentering later is not difficult. 

Feedback is quick and to the point. 

The program can be used without assistance. 



DESCRIPTION OF THE FINAL PRODUCTS 

The completed CDR games —PEGS! for PRESCHOOL and PEGS! for SECONDARY— are included in 
sleeves at the back of this report. Also included are the accompanying printed guides for users, distributed 
with each CDR. Each version contains animations of six child characters behaving in ways familiar to 
educators. The purpose of both games is to increase participation of the child characters in simulated 
learning activities. The player is asked to choose from four typical school-related activities. A practice 
session with one child character is available for a warm-up. There are options to play the game at three 
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levels of difficulty. At the easy level there is immediate feedback that flashes on the screen. At the most 
difficult level, the pacing of the children's responses increases, and feedback is delayed until the end of 
the game. The menu provides additional options for obtaining text information about each child character, 
in-depth information about the strategies, and descriptions of the behavior problems of the child 
characters. Figure 1 illustrates the menu screen for the early childhood version and Figure 2 shows the 
menu screen for the secondary version. 



Figure 1. Menu Screen of PEGS! for PRESCHOOL 




Figure 2. Menu Screen of PEGS! for SECONDARY 
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As an activity begins, the animated simulation shows children behaving in characteristic ways. 
The player selects a particular child by clicking the mouse on that child and then selecting among 12 basic 
positive management strategies with the goal of increasing or maintaining the child's participation in the 
activity. The child will then respond to the choice. If the choice meets the needs of that individual, there 
will be an increase in participation and a decrease in problem behavior. When a choice is not a good 
match, a child’s participation will drop and problem behavior increases. While the player attends to one 
child, the other children continue to engage in a range of behaviors. If ignored for too long, the others 
may exhibit negative, non-participatory behavior, and the player may lose the game. Figure 3 displays the 
strategy buttons and icons. 

Figure 3. PEGS! Strategies You Can Use* 




‘Strategies are modified for early childhood and secondary students. 

A player can explore alternative responses by repeating the same activity but choosing different 
strategies. Each time the game is played, a randomly selected small group of children — each with a 
different configuration of behavior problems — is presented. The game is won when all of the children are 
participating with enthusiasm and the player has maintained full participation of the group. At the end, the 
program provides feedback about each child's participation and an analysis of the management strategies 
used. This feedback has a print option. 
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PROJECT ACTIVITIES and ACCOMPLISHMENTS 

Table 2 summarizes project tasks, activities, and outcomes that were undertaken during the three-year 
grant period to achieve the final products described above. The specific project tasks were: 

Task 1. ©esigm and develop two C©-KOM§ in game formats using advanced computer technology. 

Task 2. Conduct formative evaluations during development of both C©-RQM programs. 

Task 3. ©evelop a plan for disseminating information about the products. 

Task 4. Conduct summative evaluation of project activities and products with specified users. 

The project was conducted as described in the original proposal, on time and within budget. The series is 
now complete. The products are unique instructional technology — fully interactive CD-ROM games for 
adults providing simulated practice using positive behavior management strategies. Table 3 summarizes the 
content contained in the preschool and secondary (middle/high school) versions. 

Information about the completed PEGS! series in available on the Internet at: 



http://www.downloadleaming.com/teachers/pegs.html 



The Parents and Professional Advisory Councils 

A diverse, volunteer Advisory Council comprised of about ten members was actively involved in all four 
project tasks each project year. The Council was reconfigured somewhat every year to represent various 
stakeholder groups, adding members as the project emphasis changed from a focus on early childhood to 
young teens. The Council members were diverse in ages, gender, ethnicity, professions, and parenting 
children with and without disabilities. Meetings were held twice a year and the members independently 
reviewed each CDR version in the development process. Council members gave valuable feedback to the 
staff about design, content, and usability features. They participated in the sequences of formative 
evaluations and tested the prototype individually for both content and technical concerns, i.e., ease of use, 
readability, and relevance. They also reviewed the final summative evaluation design and 
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Table 2 . Project PEGS! Overview of Activities and Accomplishments 
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Table 3. Project PEGS! CD-ROM Content for PC or Mac Operating Systems 



• Menu for navigating the CD-ROM options 

Learning goals and objectives for user 

Three difficulty levels for playing the game and practice option 

Additional supportive information 

Help in using the program 

Tips for winning the game 

Links to other resources 

• Twelve child characters animated in color graphics with audio responses 

Six children, ages 2-5, and six teens, ages 15-17 
Children with challenging behaviors in each CDR 
Children with typical behavior problems in each CDR 
Gender, ethnic, cultural, and age balance 
Sufficient realism for authenticity 

• Introductory text vignettes 

Text files of school history and educational recommendations for each child 
Text files of psychological and developmental histories for special needs children 

• Simulated behavioral responses by children to all strategy options in all activities 

Behavioral continuum from participation to highly disruptive behavior 
Text descriptions of behavioral responses for each child 

• Twelve positive management strategies known to increase participation in activities 

Text descriptions, expanded examples, misuses 

• Four learning activity options for early childhood version 

Circle Time 
Sand Box Activity 
Centers 

Outside Group Play Activity 

• Four learning activity options for secondary version 

Individual Academic Time 
Group Discussion 
Hallway Action 
Science Lab 

• Feedback to user 

On-going "Participation Index" to alert user to children's levels of participation 
Clock to alert user about time remaining for an activity 
Summary feedback showing frequency of each strategy used with each child 
Feedback about general management style with print option 
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instruments prior to field testing. Appendix A contains information about the Council members, agendas 
for the meetings, and summaries of their evaluative reviews. 

Task 1. Design and Develop the Two Products 

The first project activity was to prepare instructional design documents and establish learning objectives 
for users. Early childhood and secondary program design documents are contained in Appendix B. The 
resulting learning objectives for potential users shaped the content and instructional design of the CDRs. 
These objectives are: 

> Identify children whose behavior needs improving. 

> Select strategies that help children participate in learning — and keep them participating. 

> Experiment with using a range of core strategies in typical educational activities. 

> Learn to apply the strategies to children with different problem behaviors. 

Content development, technical development, graphical design, and formative evaluations proceeded on 
schedule. Opinions and advice of Council members, external professional consultants, teachers and 
assistant teachers, parents of children with and without disabilities, and other stakeholders were taken 
very seriously at each stage of the project and have contributed to the quality of the final products both in 
content and in technical design. 

Technical development. Project staff co-partnered with an experienced private firm, LetterPress 
Software, Inc. in Logan, Utah for development of the CD-ROMs. Using its proprietary computer 
simulation engine, the PEGS! software applications were developed with an instructional component 
technology called iComponents (LetterPress Software, 2002). The iComponent used in the project series 
encapsulates a knowledge base of rules (children’s behavior) with an instructional strategy that allows 
learners to act upon objects (animations based on children’s profiles) and see the consequences of then- 
actions (changes in the children’s behaviors) within a simulation-learning environment. The design is a 
practical application of Merrill’s instructional design theory (1994; IDT 2 , in press). The simulation 
activities are additionally supported by embedded textual resources with information on the children’s 
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profiles, descriptions of the potential behaviors and intervention strategies, and tips for using the program 
and a detailed user feedback mechanism. 

Behavior problems portrayed in the PEGS! simulations. Selection of typical behavior problems for 
inclusion in the CDR simulations required a review of research and existing research findings to identify a 
broad range of behavioral challenges that can interfere with children's learning and development. At one 
end of this continuum are children who have not yet been identified but are emotionally vulnerable and at- 
risk for learning or behavioral problems. Effective preventative interventions are needed for them (Kilgo et 
al, 1999; Serna, Lamros, Nielsen, & Fomess, 2002). At the other end of the continuum are children who 
have passed the screening threshold for serious problems and require carefully selected interventions (Feil, 
Walker, Severson, & Ball, 2000; Sugai & Homer, 2002; Walker, et al., 1996). Many of these children have 
internalized problems that may be overlooked. Some hesitate to participate or are fearful and anxious, 
withdrawn, dependent, or overly demanding of adults' attention. (Achenbach, 1992; Developmental 
Therapy Institute, 1996; Kaiser, Hancock, Cai, Foster, & Hester, 2000; Kaiser et al., 2002). In contrast, 
other behavior problems which may be externalized are anti-social, aggressive, self-control problems 
(Blair, Umbreit, & Bos, 1999; Epsteein, Cullinan, Ryser, & Pearson, 2002; Kupersmidt, Bryant, & 
Willoughby, 2000; Sprague & Walker, 2000; Stormont, 2001). Anticipating that users of the CDR would 
face this broad range of problem behaviors in their actual classrooms — from no manifest problems to 
highly disruptive or severely withdrawn behavior — a pool of problem behaviors was compiled for possible 
inclusion in the CDRs. 

The lists were then reviewed by the project Advisory Council composed of general and special 
education practitioners and parents of children with and without disabilities in the respective targeted age 
groups. Each final list contained the typical problem behaviors identified unanimously by Council 
members as representative of the behavioral challenges faced on a daily basis by early childhood educators 
and secondary school teachers (Table 4). 
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Table 4. Simulated Behavior Problems of Child Characters in P EGS! CD-ROMs 



PEGS ! for EARLY CHILDHOOD PEGS! for SECONDARY 



Destructive to materials or toys 
Hurtful to self or others 
Hyperactive 
Fails to listen 

Frequently unhappy, sensitive, or sad 

Irritable/physical complaints 

Short attention span, restless 

Talks or behaves like a younger child 

Temper tantrums 

Upset by others 

Withdrawn 



Destructive to materials 
Hurtful to self or others 



Listening difficulties/daydreaming 
Unhappy/overly sensitive 
Physical complaints 
Short attention span/restless 
Talks or behaves like a younger child 
Uncontrolled anger/rage 
Agitated around others 
Withdrawn/detached 



The child characters. As the project progressed, staff members, professional consultants, and the 
project Advisory Council together designed child characters and their troubling behavior problems to be 
included in the CDRs. The individual characters were designed with simulated personalities and behavioral 
patterns that would offer users the widest possible practice in a range of behavior management 
challenges — from mild to severe (Figure 4). Appendix C contains the text vignette summaries of the child 
characters that were the basis for designing the graphics and supporting text case material. 

The Council then reviewed preliminary animations portraying these behaviors to assure that they 
were culturally sensitive, with multi-ethnic representations among the child characters. Finally, each 
council member and other stakeholders ranked the portrayed behaviors from mild to severely challenging 
to confirm that the content included the range of real-life problems evident among children. (See Task 2.) 

The strategies. The 12 behavior management strategies included in the PEGS! program are 
fundamental practices familiar to educators and compatible with most educational programs — in both 
public and private settings. The strategies are divided into three groups — those that anticipate and avoid 
behavior problems, those that are recommended to keep behavior from escalating, and those to be used 
when behavior becomes highly disruptive. 
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Figure 4. Meet the PEGS! Students 
Preschool Children 




Ann Belinda Brad Carlos Cayla Leon 



Middle/High School Students 





Jameel Victor 





BEST COPY AVAILABLE 
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These core strategies were adapted to the targeted age groups for each CDR (Table 5). 



Table 5. Management Strategies Included in PEGS! CD-ROMs 



PEGS! for PRESCHOOL 



PEGS! for SECONDARY 



Strategies to Anticipate and Avoid Problems 



Encourage and Praise Encourage and Praise 

Motivate with Materials and Activities Motivate with Lessons and Content 

Organize Materials for Child -- 

Describe Procedures Explain Procedures 

Model Expected Actions Model Positive Relationships 

Strategies to Keep Behavior Problems from Escalating 



Move Closer to Child 
Redirect to Activity 
Reflect Positive Words and Actions 
Remind Child of Rules 



Signal Awareness 
Redirect/Refocus on Task 
Reflect Confidence in Student 
Remind Student of Rules 
Interpret Student’s Words and Actions 



Strategies to Control Highly Disruptive Behavior 

Time-Out in Room 
Hold Firmly 
Remove from Room 



Suggest Student Take Time-Out in Room 

Confront/Reprimand 

Remove Student from Room 



Note. Definitions of these strategies were specified during the design phase and are included in Appendix B. 



These strategies have been used effectively in fully inclusive, partially inclusive (pull-out), self- 
contained, and home-based programs with the intended age groups of children (Wood, Davis, & Swindle, 
1998). The strategies were originally selected after an extensive review of theories and research about 
effective management strategies that promote responsible behavior in children and youth (Beyda, Zentall, 
& Ferko, 2002; Blair, Unbreit, & Bos, 1999; Bredekamp, & Copple, 1997; Dwyer, Osher, & Hoffman, 
2000; Kamps, Tankersley, & Ellis, 2000; Martens, Peterson, Witt, & Cirone, 1986; Quirk, 1993; Wolery & 
Bredekamp, 1994; Wolery & Sainato, 1996; Wood, 1996). 

Task 2 Formative Field Testing 

Formative evaluation activities were concerned with issues of content validity, user friendly format, 
market need, instructional effectiveness, practicality, technical quality, and potential usefulness. Because 
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it was essential that the product be educationally sound, instructional ly effective, and relevant to the 
specified stakeholder groups, formative evaluation was essential in all three project years. Representatives 
of the identified stakeholder groups included: Novice teachers; paraprofessionals; student teachers; school 
administrators and supervisors; school support staff including psychologists, social workers, and 
counselors; parents of children with and without disabilities; teacher educators; community service 
providers; and state level educators. Formative evaluation activities and outcomes are summarized below. 

Formative evaluation question 1 . Are all of the troubling behaviors specified in the original 
proposal included among the behavioral descriptions of the child characters in the simulations, including 
those children with and without special needs? 

To answer this question. Council members and program staff with expertise in behavior disorders and 
early childhood development were asked to read the descriptive vignettes of the child characters designed 
for each CDR. They independently matched the vignettes with the list of targeted problem behaviors 
identified in the original proposal. The results verified that all 1 1 targeted behaviors for preschool (Table 
6) and the 10 behavior problems included in the secondary version were portrayed in the descriptions of 
the child characters for the respective versions (Table 7). 

Formative evaluation question 2 . Do the child characters portray a range of children's problem 
behaviors from most severe to least severe? 

Staff experts and Council members were then asked to rank the severity of the children's behavior 
problems from most severe (ranked 1) to least severe (ranked 6) to verify the authenticity in range of 
behavioral severity conveyed by individual child characters who had been designed to represent those 
with and without social, emotional, or behavioral problems. The experts and Council members were 
unanimous in correctly identifying those with special needs in both versions — those with the most severe 
behavior problems and those with the least severe problems. Among the early childhood characters 
(Table 8), expert rankings only differed from Council rankings for a shy child without externalized 
behavior problems (Carlos, with English as a second language). Council members saw him as a greater 
potential problem than a child with externalized, immature behaviors (Belinda). 
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Table 6. Frequency of 11 Behavior Problems Identified in Vignettes for PEGS! for PRESCHOOL by 
Council Members and Expert Panel 



Behavior Problem 


Council 


Experts 




(n = 7) 


(n = 7) 


Withdrawn from peers or adults 


20 


21 


Short attention span 


15 


11 


Talking or behaving like a younger child 


14 


12 


Frequently unhappy, overly sensitive, or sad 


13 


7 


Uncontrollable temper or tantrums 


11 


11 


Agitated around others 


11 


6 


Impertinent, defiant, resentful, or negative 


10 


9 


Irritable 


9 


8 


Hurtful or destructive to self or others 


9 


7 


Hyperactive 


8 


6 


Listening difficulties 


5 


1 



Table 7. Frequency of 10 Behavior Problems Identified in Vignettes for PEGS! for SECONDARY by 
Council Members and Expert Panel 



Behavior Problem 


Council 
(n = 11) 


Experts 
( n = 10) 


Withdrawn from peers or adults 


25 


30 


Frequently unhappy, overly sensitive 


25 


29 


Hurtful or destructive to self or others 


25 


21 


Agitated around others 


17 


13 


Physical complaints 


17 


10 


Impertinent, defiant, negative 


16 


12 


Listening difficulties/daydreaming 


16 


18 


Short attention span 


16 


19 


Uncontrollable anger/rage 


13 


10 


Talking or behaving like a younger child 


11 


11 



Among the students in the secondary version (Table 9), the rankings again varied slightly for children 
with internalized (Jameel and Victor) and externalized behaviors (BB and Beth). While these findings 

of relative severity of 
offer practice in a range 



indicate that Council members and experts differed slightly on their views 
externalized versus internalized behavior problems, it is evident that both CDRs 
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behavior problems. The rankings also suggest that there are clear differences in the severity of children's 
problems that can be detected by professionals and other stakeholders alike. 



Table 8. Rankings in Severity of Problem Behaviors in Early Childhood Version 



Early Childhood 
Characters 


Council Average 
Rank Order 

n = 7 


Expert Average 
Rank Order 
n = 6 


Brad 3 


1.0 


1.2 


Leon 3 


2.5 


2.0 


Cayla 3 


2.5 


2.8 


Belinda 


4.0 


4.8 


Carlos 


5.0 


4.2 


Ann 


6.0 


6.0 


Note: Most severe = 1 , least severe = 6 
‘Children with the most challenging behaviors 

Table 9. Rankings in Severity of Problem Behaviors in Secondary Version 




Secondary School 


Council 


Expert 


Characters 


Average Rank Order 


Average Rank Order 




n= 11 


n = 10 


BB 3 


1.5 


2.0 


Jameel 3 


1.8 


1.2 


Caron 3 


2.7 


2.8 


Beth 


4.1 


4.8 


Victor 


4.8 


4.2 


Aleesa 


6.0 


6.0 



Note: Most severe = I , least severe = 6 
‘‘Children with the most challenging behaviors 
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Formative evaluation question 3 . Are the strategies included in the PEGS! series known to 
potential users and viewed as effective ways to deal with behavior problems? 

Before beginning field tests for the PEGS! for SECONDARY version, 27 teachers and assistant teachers in 
local middle and high school programs were asked to rate how frequently they used each of the 12 
strategies that had been selected for inclusion in the program (Table 10). All strategies were rated as used 
at least "sometimes" (>3.0), with Provide Encouragement most frequently identified by high school 
educators (average 4.4); and Model Positive Relationships (average 4.7), by middle school educators. 
However, those teaching middle school students reported a generally higher frequency in using the 
strategies than those teaching high school students. There were also differences in their reports of least 
frequently used strategies, with high school educators identifying Ask Group/Individual to Review Rules 
(average 3.1), while the middle school educators identified Suggest Voluntary Time-Out (average 3.1). 

A question concerning their use of ignoring was also included. This was consistently identified by 
both groups of educators as used least often and "seldom effective." This finding was consistent with the 
approach taken in the CDRs to encourage educators to respond to students with troubling behaviors using 
positive management and instructional interventions rather than ignoring. 

This information confirmed that the 12 basic strategies selected for inclusion in the PEGS content 
were known and used by both middle school and high school educators. While both groups consistently 
viewed the strategies as "sometimes" effective, strategies that they reported as using most frequently were 
not the ones that they rated as most effective. It was anticipated that improvement in skills with which 
they used the strategies would also increase effectiveness. 

Formative evaluation question 4 . Are the child characters' behavior simulations authentic true- 
to-life responses? 

To verify the authenticity of the simulated behavioral responses to every possible strategy that might be 
selected by a user, an expert panel was established for each version. The experts each had 20 to 35 years 
experience in classrooms for troubled children as teacher trainers, program supervisors, and direct service 
providers. They independently reviewed the text vignettes, behavior descriptions, and program 
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Table 10. Educators' Familiarity with Management Strategies Selected for Inclusion in the 
PEGS ! CD-ROM for Secondary 





How frequently do 


How effective is 


' 


you use this 


this with the 




strategy? 


a 


students you 
teach? 6 


CD-ROM Content 




High 


Middle 


High 


Middle 


Strategies to Anticipate and Avoid Problems 


School" 


School* 1 


School 


School 


Provide Encouragement 


4.4 


4.6 


3.3 


3.8 


Motivate with Content/Lessons 


3.8 


3.8 


3.6 


3.2 


Explain Procedures 


4.3 


4.4 


3.7 


3.7 


Model Positive Relationships 


4.2 


4.7 


3.6 


3.5 


Strategies to Keep Behavior Problems from Escalating 


Signal Awareness 


3.7 


3.9 


3.2 


3.2 


Redirect/Refocus on Task 


4.3 


4.5 


3.2 


3.6 


Reflect Confidence in Student 


4.3 


4.5 


3.4 


4.1 


Ask Group/Individual to Review Rules 


3.1 


3.6 


2.8 


3.2 


Connect Actions to Feelings (Interpretation) 


3.6 


3.9 


3.4 


3.2 


Strategies to Control Highly Disruptive Behavior 


Suggest Voluntary Time-Out in the Room 


3.8 


3.3 


3.2 


3.U 


Confront Unacceptable Behavior 


4.2 


4.4 


3.7 


3.7 


Remove From the Room 


3.4 


3.6 


3.5 


3.5 


Ignore Students' Behavior 


2.8 


3.1 


3.0 


2.4 



a 1 = never, 2 = seldom, 3 = occasionally, 4 = often, 5 = consistently. 
b 2 = never, 2 = hardly ever, 3 = sometimes, 4 = usually, 5 = always. 
c High school educators n = 10. 
d Middle school educators n = 17. 



specifications (computer rules) governing the behavioral response(s) that each child character would 
mostly likely make to every possible strategy. Panel members then met and compiled the selections. The 
standard for authenticity was consensus. This process resulted in a total of 6,336 individualized computer 
rules developed by consensus agreement among the experts (11 behavioral responses to each of 12 
strategy options for each of the 12 children in the 4 different activities). The results of this process 
provided assurance that the portrayed responses by the child characters to the strategies in the simulations 
are valid and authentic to real-life situations for young children and teens. 



ERiC 



25 



Project PEGS! CD-ROM, Page 19 



Formative evaluation question 5 . In the preliminary (Beta) versions are content, instructional 
design, and technical attributes interesting, user friendly, and perceived as valuable by potential users? 
With completion of the Beta version of PEGS! for PRESCHOOL, a group of 16 college students taking a 
behavior management course at Georgia College and State University was asked to use and evaluate the 
program over a 10-day period. They then rated the content, value as a learning experience, and technical 
attributes on a 26-item checklist (1 = highly satisfactory to 5 = not satisfactory). They also participated in 
a focus group discussion. Results indicate that they were generally approving of the PEGS! program as a 
learning tool, found it reasonably user friendly, and valuable for student teachers. Chart 1 contains a 
summary of their ratings. Appendix D contains characteristics of the group, a copy of the rating checklist, 
the focus group questions, and a summary of the focus group discussion. 



Chart 1. Average Ratings of PEGS! for PRESCHOOL by 16 College Students 




Content Learning Ease of Use Total 



Note: Scale of 5 = agree strongly to 1 = disagree strongly 

When the Beta version of the PEGS! for SECONDARY was completed, a group of 41 college 
junior and senior students from the Department of Occupational Studies in the College of Education at the 
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University of Georgia were asked by their instructor to use and evaluate the program. All of the students 
are planning to work in educational settings where they will interact with middle and high school 
students. The study was conducted by Emise S. Singleton (2002), a doctoral student in the Department of 
Instructional Technology in the College of Education. Chart 2 illustrates that the majority of the students 
agreed that the CDR has realistic content (average rating =3.9), provides ample learning opportunities 
(average = 4.1), is easy to use (average = 3.6), and could be useful for middle and high school teachers 
(average = 4.1). Appendix E contains their ratings for each item. 

Chart 2 Average Ratings of PEGS! for SECONDARY on 21 Criterion Items by 41 College Students 
for Formative Evaluation 

4.2 

4.1 
4 
3.9 
3.8 
3.7 
3.6 
3.5 
3.4 

3.3 

Content Learning Ease of Use Total 

Note: Scale of 5 = agree strongly to 1 = disagree strongly 




As a follow-up with the PEGS! for SECONDARY version, two Master's degree students in the 
Department of Instructional Technology were asked to participate in user interviews after evaluating the 
program for two weeks. They provided their perspectives on instructional design and clarity, cosmetic 
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design, and program functionality. Chart 3 illustrates their average levels of satisfaction in these three 
areas. According to these reviewers, the program interactions met the goals of the program. They reported 
that the overall design was effective and the objectives were clear and easily achieved by interacting with 
the CD-ROM. One of the reviewers made the following comment: 

The interactions do seem appropriate for the objectives, but the design 
needs a little work on the details of the presentation. I found the strategy 
icons difficult to remember and when I selected one, the delay between 
selection and hearing the response it represented was significant and 
made it difficult that while I was waiting to see if that worked or not, that 
another student would present a problem and there was nothing I could 
do about that one while I was working on the first one. 

[Note: This was an intended consequence of ignoring children by a user.] 



Chart 3. Average Ratings of PEGS! for SECONDARY on 10 Design Criteria 
by 2 Independent Instructional Design Experts 




Note: Scale of 5 = strongly agree to 1 = strongly disagree 
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Reviewers rated cosmetic design the highest. They reported that graphical designs were easy to 
understand and the colors enhanced the visual quality of the program. The one recommendation for 
improvement was to include directions on how to begin the program: 

It took me a little while to understand what I was supposed to do. It 
might be good to have some kind of splash page or something that gave a 
brief description of what the purpose of the [activity] was and how to use 
it. 

[Note: This recommendation resulted in enhanced entry in the final product designs. Comprehensive 
reports of these interviews are in Appendix E.] 

Findings and recommendations from each of these formative evaluation activities were reviewed 
by project staff and LetterPress Software designers. Revisions were made when changes would clearly 
enhance the final product and were possible within the scope of the budget. Overall, the results of these 
numerous formative evaluation activities contributed immeasurably to the quality of the final products in 
meeting the standards set for content validity, interest, relevance, usefulness, and user friendly design. 



Task 3. Planning for Information Dissemination and Product Distribution 

The foundation for planning dissemination activities of this PEGS! project was set by the encouraging 
field test results with the prototype version for elementary school teachers, developed during 1999 by the 
Georgia Department of Education,. The Georgia results indicated that the product was effective in 
increasing behavior management skills of elementary school teachers in their own classrooms and 
reducing the instances of students’ problem behavior by 58% (Georgia Department of Education, 2000). 
Subsequently, a series of meetings was held with the Georgia State Improvement Grant (SIG) project 
team and steering committee to plan for a statewide distribution system of the PEGS! elementary school 
version. Starting in January 2001, the project staff and John O’Connor with the Georgia Department of 
Education met with coordinators of the Georgia Learning Resource Systems (GLRS) who are responsible 
for staff development in each of the 17 educational regions of the state. They were provided an 
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introduction to the CDR and a brief opportunity to use it independently. They were then asked to advise 
project staff as to the potential usefulness of PEGS! as a tool for inservice education. The responses were 
highly positive. 

The next step was to ask each regional coordinator to sponsor a local meeting and invite 
principals and special education coordinators to use the CDR in a computer lab. Three such sessions were 
held in different geographic areas of the state, with a total of about 90 leadership individuals participating. 
From this group, a statewide information and training system was developed in conjunction with the 
Georgia Department of Education GLRS system. (See Supplemental Information at the end of this 
report.) 

A statewide Georgia distribution arrangement was established with LetterPress Software, Inc. in 
May 2001 and there were steady requests for PEGS! for ELEMENTARY through their Internet subsidiary 
www.downloadleaming.com . Two versions were available: (1) an individual license for single computer 
use which included the CD-ROM and a brief study guide as an introduction about how to use the 
program; and (2) a school/institution site license that permits downloading the software to a single 
building for an unlimited number of users. Multiple copies of the printed study guide were included. The 
site license was specifically designed to meet the needs of colleges, universities, technical schools and 
other institutions where computer labs are available for adult learning. A letter describing the Georgia 
phase of distribution is included in Appendix F, 

National marketing plan: From this Georgia pilot testing, a national Information Dissemination 
and Distribution Plan was designed for the entire PEGS! series (Figure 5), and an informational flyer was 
designed for professional conferences, mailings, and as advertisements for professional journals (Figure 
6). Six potential user groups were identified: Individuals, personnel in public education programs, 
personnel in private schools, prospective educators in community colleges and technical schools, those 
enrolled in teacher education programs in colleges and universities, and mental health service providers. 
To make both information and the products available to this broad market, an array of vehicles for 
dissemination and distribution was selected. 
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* Select preschool, elementary, or high school — 

* Choose among various activities 

* View simulations of problem behaviors 

* Analyze what's needed 

* Experiment with 12 practical intervention strategies 

* See animated reactions of students to your choices 

* Receive feedback 

» Increase your own skills and decrease problem behaviors- 
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contact tine Developmental Therapy-TeacMng Programs, 



Tlhe University of Georgia, at 7@6-369-5(68>9 or www.nga.edua/ dtttp 
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These included: . 



Dissemination Activities 

• The Internet 

• State departments of education 

• Research institutes 

• Publishers and distributors of 
educational materials 

• Private foundations interested in 
education 

• US Department of Education grant 
funded programs 

• Head Start staff development and 
training programs 

• The Special Education Technical 
Assistance and Dissemination 
Network 



Dissemination activities accomplished to date. Major sources for information dissemination have 
been through four sources: the Developmental Therapy-Teaching Programs (DTTP) website at the 
University of Georgia: www.uga.edu/dttp : the LetterPress website: www.downloadLeaming.com; the 
DTTP newsletters with a mailing list of 500 educators, administrators, and teacher educators; and the 
DTTP Trainers Network with 1 8 certified regional trainers and 9 certified national instructors providing 
technical assistance and staff development activities to schools and other child serving agencies 
nationwide. 

In 2001, the Beta version of PEGS! for PRESCHOOL was presented at the Georgia Association 
for the Education of Young Children. That same year, 500 preliminary information flyers were 
distributed, and PEGS! was demonstrated at two annual international conferences: The CCBD conference 
strand, Integrating Technology in Program Development for Children and Youth with E/BD, and the 
National Association for the Education of Young Children. PEGS! was also presented at the annual 
Georgia Psychoeducational Network Conference at St. Simons, Georgia. 

In 2002, 517 information flyers were distributed, and PEGS! was introduced at five national 
conferences: The Future of Learning for all Students (national technology demonstration event); the 
annual research project directors meeting sponsored by the U.S. Office of Special Education Programs; 



Sources for Dissemination 

Professional associations and 
sponsored conferences 
Professional journal publications 
and advertisements 
University colleges of education 
The Developmental Therapy 
Institute newsletter 
The Developmental Therapy 
Regional Trainers Network 
ERIC 
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the annual conference of CCBD; the bi-annual International Child and Adolescent Conference sponsored 
by the Behavioral Institute for Children and Adolescents; and the 15 th annual research conference 
sponsored by the Research and Training Center for Children's Mental Health, University of South Florida, 
in Tampa. The PEGS! products were well received at these demonstrations, and requests were received to 
obtain copies. Additional demonstrations of the programs were provided to schools and direct service 
agencies that were receiving technical assistance and staff training through the Developmental Therapy- 
Teaching Programs, including training childcare workers in Washington State, and to administrators at 
family mental health centers in Florida. 

The first series of direct mailings and journal advertisements are planned for release in March 
2003, to coincide with completion of the final revisions to the series. Advertising has been arranged for 
Teaching Exceptional Children. Informational flyers and program advertisements will also be distributed 
at the annual international conference of the Council for Exceptional Children in April 2003 (500 copies), 
and the annual conferences for Council for Children with Behavioral Disabilities and the Division for 
Early Childhood in October 2003 (500 copies). 

Product distribution activities. Because of the broad range of potential users — preservice student 
teachers, general and special educators, direct service providers in many related child-serving programs — 
a decision was made to avoid a single source distribution agreement. By using multiple suppliers, a broad 
distribution market is anticipated. Each distributor will negotiate directly with LetterPress Software to 
establish a distribution agreement. To date, 95 individual licenses and 13 site licenses have been 
distributed through < www.downloadlearning.com >, in addition to the complimentary copies provided 
in Georgia. Complimentary copies are being offered to State Department of Special Education Directors. 
The final, revised versions for all three CDRs in the series will be available at the same website on March 
1, 2003. The products are currently also available by catalog and Internet through The Guidance Channel. 
Marketing possibilities were also discussed with five other distributors, requiring many weeks or months 
with in-house reviews. While all had praise for the products, they finally declined distribution either 
because PEGS! did not fit their current marketing focus or because they did not have the dollars in their 
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budget to promote the products at this time. In addition, five publishers and distributors are currently 
interested in the products and have them under review, as indicated above in Figure 5. 

Task 4. Project Evaluation 

The ultimate value of the PEGS! CDRs is in their potential to actually help teachers and assistant teachers 
who are having problems with disruptive, negative children in their classrooms. With this in mind, plans 
for field tests were developed, and standards for effectiveness were established (Figure 7). For both 
products, the standards were the following: First, potential users should find the CDR to have relevant 
content, provide a helpful learning experience, and be used with ease. Second, there should be evidence 
of positive carryover from using the CDR into actual classroom practices. 

The field tests were conducted in May-July 2001 for the preschool version and in October- 
December 2002 for the secondary school version. The same instruments and data collection procedures 
were used in both sets of field tests, with the exception of modifications in the skill rating instrument 
adapted to the variations in behavior management strategies recommended for the two age groups. The 
field test sites also differed in the way they provided services to children with challenging behaviors. The 
early childhood programs were fully inclusive, and the secondary sites were fully self-contained special 
education. Table 11 contains characteristics of participants in the field tests for each version. They are 
described in greater detail below. 

Participants in PEGS! for PRESCHOOL Field Tests 

Early childhood program directors were contacted about the proposed field testing at three locations — a 
demonstration school at the University of Georgia, a private childcare facility, and a Head Start program. 
All three programs included children in the 2- to 5-year-old age range. At these sites, 24 early childhood 
educators volunteered to use and evaluate the CDR. The group included 11 teachers and 13 assistant 
teachers. Their average experience working with children was 8 years (range =1-25 years). Each 
participant had access to a computer with printer, sound, and CDR capability. 




35 



Project PEGS 7 CD-ROM. Page 29 




co 

e 

•2 

<3 

1 



Co 

.j§> 

s 

5 

a 

s: 



A 

t*. 




36 



Project PEGS! CD-ROM, Page 30 



Table 11. Characteristics of 43 Educators Volunteering to Field Test PEGS! CD-ROMs 





Early 
Childhood 
Educators 
(n = 24) 


Middle/ 
High School 
Educators 
(n = 19) 


GENDER 


Female 


21 


8 


Male 


2 


11 


Unreported 


1 


0 


RACE 


European American 


10 


12 


African American 


13 


5 


Asian 


1 


0 


Multiracial 


0 


1 


Unreported 


0 


1 


AGE 


Under 20 Years 


1 


0 


20 -29 Years 


9 


5 


30-39 Years 


6 


8 


40-49 Years 


5 


3 


50 - 59 Years 


2 


2 


60 — 69 Years 


0 


1 


Not reported 


1 


0 


CURRENT POSITION 


Teacher 


11 


12 


Teaching Assistant 


13 


7 


EXPERIENCE WITH 
CHILDREN 


Average in Years 


8.5 


6.9 


Range 


1 -25 


1 -35 


COMPUTER SKILL 


Highly Skilled 


3 


3 


Moderately Skilled 


16 


14 


Not Very Skilled 


5 


2 


HIGHEST DEGREE 3 


High School 


2 


4 


Associate Degree 


2 


2 


Bachelor Degree 


5 


3 


Graduate Level 


3 


10 


CHILDREN TAUGHT 


Average Class Size 


22 


7 


Range 


12 -40 b 


3-8 


Total Children 


393 


118 



3 Incomplete data from early childhood participants. 



One participant reported separate morning and afternoon programs of 20 children in each group. 
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In the demonstration school, enrollment for each class was balanced for children’s age, gender, 
ethnicity, and socioeconomic status. The private childcare facility took children on a space-available 
basis, and the Head Start program used national eligibility standards for enrollment. The programs 
were licensed by the state and maintained state standards for materials, schedules, activities, safety, 
nutrition, and adult-child ratios. The average number of children in a classroom was 20 with two 
adults. The programs were well regarded in the community among parents of young children, and the 
personnel in each program were knowledgeable about developmentally appropriate early childhood 
practices. Administrators encouraged staff to participate in the field test. An incentive for participation 
was a complimentary copy of the CDR for each school. 

All three early childhood sites had policies of inclusion of children with disabilities. In the 
demonstration program, seven participants reported teaching children with identified disabilities. The 
private childcare program had one child identified with special needs, and no children were reported as 
having an identified disability at the Head Start program. At the time of the study, none had children 
specifically identified with an emotional or behavioral disability; however, the staff at each program 
indicated informally that they had children with challenging behaviors. 

Early childhood educators in the classroom carryover study. To explore possible carryover from 
CDR to actual classroom practice, 14 of the 24 early childhood volunteers agreed to participate in further 
field testing. While slightly less experienced on average than the larger group of 24, these participants 
were otherwise representative: 8 European Americans, 5 African Americans, and 1 Asian American; 12 
females and 2 males; 9 teachers and 5 assistant teachers. They averaged 6.2 years experience (range = 1- 
12 years). They understood that they would be observed for 30 minutes while working with children 
before and again after using the CDR at a time and day they selected, with activities of their own 
choosing. The only stipulation was that they select the same activity(ies) if possible for both observations. 
They were also assured that neither they nor the children would be identified in the observations. 
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Participants in the PEGS! for SECONDARY Field Tests 

Principals were contacted in local area middle and high schools providing special education services to 
young teenage students with emotional and behavioral disabilities. Administrators encouraged teachers 
and assistant teachers to volunteer to field test the CDR. An incentive for participation was a 
complimentary copy of the CDR for the school. Volunteers were obtained at two locations — a rural 
middle school for grades 6, 7, and 8, and a psychoeducational center serving middle and high school 
students with severe social, emotional, or behavioral disabilities. The middle school provided both 
inclusion and resource room (pullout) special education for students with mild to moderate learning or 
behavioral disabilities. The psychoeducational program provided part-day, self-contained special 
education. 

There were 19 volunteers, all working in their respective special education programs; one also 
taught physical education with all students in the middle school. Of these, 13 taught middle school-aged 
students (grade 6-8) and 6 taught those in high school (grades 9 - 12). The group included 12 teachers 
and 7 assistant teachers. Their average teaching experience was 6.9 years (range =1-35 years). Five had 
experience teaching in general education, and one had taught in an alternative school. Each participant 
had access to a computer with printer, sound, and CDR capability. 

Each participant reported teaching general education content areas with an average of 7 students 
in each of their daily classes. All of their students had been identified with emotional or behavioral 
disabilities, learning disabilities, attention deficit disorders, or attention deficit hyperactive disorders. 

Middle and high school educators in the classroom carryover study. To explore carryover from 
using the PEGS! for SECONDARY into actual classroom practice, 11 of the 19 participants volunteered 
for the follow-up evaluation in their classrooms. Of these, 8 were middle school teachers and 3 were high 
school teachers; 5 males and 6 females. Unlike the larger samples of participants, there were no assistant 
teachers and no African Americans who volunteered for this follow-up. All had completed a bachelor's 
degree, 6 had achieved the Master's degree, and 4 were currently completing an advanced degree. They 
averaged 6 years teaching experience (range = 2-35 years). They received the same procedural directions 
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for using the CDR, keeping logs, and selecting the classes and times to be observed. They were also 
assured that neither they nor the students would be identified in the observations. 

Instruments and Procedures for the Field Tests 

Instrument for rating the CDR. A revised version of the 24-item rating scale developed for the 
formative evaluations was again used to assess the extent to which the program met the specified 
standards for relevant content, helpful learning experience, and ease of use. For the secondary program 
evaluations, one additional item was added. Would you recommend this program to others working in a 
similar position? (Appendix G). Items were rated on a scale of 1 to 5, where 5 = agree strongly, 4 = 
agree, 3 = neutral, 2 = disagree somewhat, and 1 = disagree strongly. Ratings were summed to obtain 
average scores for each item, the three categories, and an overall rating. Criteria for effectiveness ratings 
were set at >3.0 for each category. 

Procedure for rating the CDR Participants were given individual copies of the CDR and a 
logbook. They were asked to use the CDR independently as frequently as their free time allowed during a 
2-week period and were advised that they would need at least several times alone to play and re-play the 
game. They were also asked to keep a record of activities they selected, the names of the child characters 
in the groups presented, levels of program difficulty chosen, and any text material used. The logbook 
pages contained a format for recording the date of each entry and time spent on the program. Participants 
were encouraged to make suggestions or comments reflecting their experience with the program. At the 
end of the two weeks, they rated the CDR using the rating form described above to evaluate its potential 
usefulness and effectiveness. 

Procedures for classroom follow-up evaluations. Two pairs of independent observers were used 
to obtain pre-CDR classroom performance data during a 30-minute observation in each participant's 
classroom. One observer used a time sampling procedure to record the frequency of strategies observed 
and the responses of students to the adult's strategies (or to ignoring by the adult), in 1 -minute intervals. 
At the same time, the second observer used the Adult Rating Scale, described below, to rate the skill with 
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which the participant applied the strategies selected. The participants were then given the CDR and 
logbook for independent practice. The same two observers repeated the process two weeks later for post- 
CDR observations. Details are provided below. 

Time sampling procedure. To obtain evidence of carryover into classroom practice, time-samples 
were collected both before and after participants used the CDR. This provided information about how 
frequently they used the strategies contained in the program and how the students responded to the 
participant's strategies. Every strategy and corresponding response by a student was recorded, minute-by- 
minute, during the 30-minute observation. A student's positive response was defined as any increase or 
maintenance of participation in the activity specified explicitly or implicitly by the adult. A response was 
coded as negative when a student’s actions or words were (a) disruptive to participation by self or others, 
or (b) indicative of nonparticipation. Passive participation, such as watching an adult or peer to determine 
how an activity should be done or simply waiting for a turn, was coded as positive participation. But if a 
student then became disengaged because of a long wait for adult guidance, the response would be 
recorded as a negative response. If a student was sitting quietly but clearly not involved, or if a student 
chose an activity not authorized by an adult, the response was recorded as negative. 

In addition to information about the use of strategies, the procedure included a count of instances 
of participants’ ignoring of students and the students’ responses, either positive or negative. If a student's 
words or actions indicated a need for interaction with the adult and the adult did not respond (in the 
judgment of the observer) the student's response was coded as negative. When the student's participation 
increased, ignoring was recorded as positive. Although ignoring can be a deliberate choice by an adult or 
an unintended oversight, no attempt was made to differentiate between intended and unintended ignoring 
in the time sampling procedure. By definition, planned ignoring that was effective would result in a 
positive response from a student, and therefore would not affect the total frequency of negative student 
responses. If neither a strategy nor ignoring was observed during a minute, no entry was made. The 
frequency of strategies used, instances of ignoring, and negative student responses were then totaled for 
each participant. 
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Assessing participants* skills. An observation rating scale, Rating Adult Strategies , was used to 
measure participants’ skills in applying the strategies in their classrooms. The items were qualitative 
descriptions of the 12 basic strategies contained in the CDR. Two forms were developed — one to use 
when observing early childhood educators and one for those in middle or high school programs 

(Appendix H). 

Each item was scored to indicate if the adult demonstrated the practice in a generally positive (p) 
or negative (n) manner. The coding then paired students' responses to that practice, coded as maintained 
or increased positive participation (P) or as a negative response without participation (N). Thus, each item 
had a dual rating that paired the adult's manner when applying the strategy with the student's response. 
Each item was rated and received a weighted score that focused on positive participation as the priority, 
as follows: 

5 = positive manner/positive student participation 

4 = negative manner /positive student participation 

3 = positive manner/negative student participation 

2 = item observed but unnecessary to increase student participation 

1 = item used inconsistently 

0 = item not observed 

-5 = negative manner/negative student participation 

For example, a strategy used in a positive manner by the adult resulting in a student's positive 
response would be coded p/P and receive a score of +5, while a practice demonstrated in a negative 
manner resulting in a student's negative response would be coded n/N with a resulting score of -5. 

A participant’s total skill score was the sum of the weighted rating options for all items divided by 
the frequency of strategies observed from the time sampling, to obtain an average skill score for the 
strategies actually demonstrated by the individual. This adjusted score was necessary for pre- to post- 
comparisons because all of the strategies were not always observed or needed during the repeated 30- 
minute observations. 

Reliability of procedure. The observers were not involved in the project to develop the PEGS! 
CDR. They had extensive experience in educating children with emotional and behavioral disabilities and 
each had more than 25 years experience in general and special education as program supervisors or as 
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early childhood program specialists. Prior to the field tests, each observer had achieved interrater 
agreement of greater than 90%, item-by-item, on the instrument from which the Adult Rating Scale was 
abbreviated (See Appendix H). Because observation and data collection require reliable rater judgments, 
operational terms were defined and rating protocols were reviewed in detail before the observers visited 
the classrooms. The observers also received a 2-hour training together, where they viewed videotapes of 
other typical early childhood classrooms and practiced using the coding protocols for accuracy. 

During the studies, split half reliability data were collected for estimating internal consistency in 
time sampling. Mean frequency of pre-CDR strategies used by early childhood educators recorded during 
the first 1 5 minutes of observation were compared with the mean frequency recorded during the second 
1 5 minutes (M 1 = 24.36, SD = 1 7.74; M2 = 2 1 .64, SD = 6.72; t = .55, p = .59, two-tailed). This lack of a 
statistically significant difference between the means suggests that the time sampling procedure had 
measurement stability during the 30-minute observations. The procedure was repeated for the field tests 
of the secondary version with similar results, increasing confidence that the measurements had internal 
consistency. 



Results 

Summative evaluation question 1: Do educators find the CDRs to have relevant content, provide 
a helpful learning experience, and be used with ease? 

Table 12 contains a summary of participants' ratings of the PEGS! CDRs. The 24 early childhood 
teachers and assistant teachers who volunteered to evaluate PEGS! for PRESCHOOL gave it high ratings 
of approval overall (average 4.4, where 4 = meets standard very well). Similarly high ratings were given 
for each category: relevant content (4.1), a helpful learning experience (4.4), and easy to use (4.7). 
Similarly, the middle and high school educators gave PEGS! for SECONDARY generally high ratings of 
approval overall (average 4.1), for relevant content (4.1), and for easy to use (4.1). However, they gave 
somewhat lower ratings for a helpful learning experience (3.9) although this rating also exceeded the 
acceptable minimum standard. It is interesting to note that the range of ratings by the middle and high 
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school teachers was considerably greater than those of the early childhood educators. They appeared to 
either strongly agree or to disagree somewhat, as evidenced by the ratings as low as 2. 1 and as high as 
5.0. This finding may be a result of the small sample size; or it may reflect the generally higher 
educational levels of this sample of secondary teachers. It is also possible that the highly specialized field 
in which they were working influenced their ratings. However, these field tests indicate that overall, the 
PEGS! products have achieved the standards established in the original project proposal for user 
satisfaction. 



Table 12. Average Ratings of PEGS! CDRs by Early Childhood and Middle/ High School Educators 



User Satisfaction 





Relevant 

Content 


Helpful 

Learning 


Easy to 
Use 


Total b 


Early Childhood Educators (n = 24) 
Average Rating 
Range 


4.1 

3. 5-4. 8 


4.2 

3. 7-4.7 


4.7 

4.0-4. 7 


4.4 

4.0-4. 7 


Middle School Educators (n = 13) 
Average Rating 
Range 


4.1 

2.5 -5.0 


3.9 

2.7 -4.9 


4.3 

2.9 -5.0 


4.1 

3.3 -4.9 


High School Educators (n = 6) 
Average Rating 
Range 


4.1 

3.8 -4.5 


3.9 

3.2 -4.2 


3.9 

2.1 -4.6 


3.9 

3.0 -4.4 


Total (N = 19) 
Average Rating 
Range 


4.1 

2.5 -5.0 


3.9 

2.7 -4.9 


4.1 

2.1 -5.0 


4.1 



a 5 = Agree strongly, 4 = agree, 3 = neutral, 2 = disagree somewhat, 1 = disagree strongly. 



Minimum acceptable standard was >3.0 for each category and for total. 
Total of 24 criterion statements. 
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Summative evaluation question 2 : Is there evidence of positive carryover from using the CDRs into 
actual classroom practices? 

Results of the field tests indicate that there was significant improvement in the skill with which both 
groups of educators applied the CDR intervention strategies in their classroom. After using the PEGS! 
CDRs for about 2 to 3 hours over a two- week period, 83% (10) of the 12 early childhood educators 
improved their skill scores, and 82% (9) of the 1 1 teachers in middle and high school improved their skill 
scores. There were comparable gains in mean skill scores for both groups. The early childhood 
participants increased their skills in applying the CDR intervention strategies by 41%, from a mean of 
3.3 1 to 4.67. Middle and high school educators also increased their skills by 42%, from a mean skill score 
of 2.49 to 3.53 (Chart 4). 

Chart 4. Changes in Average Behavior Management Skill Scores 
of Educators in a Two-Week Interval 

4 . 

3 . 

2 . 

1 . 

0 . 



Among the early childhood participants, the frequency of their intervention decreased 30%, from 
a total of 644 to 454 interventions. Total negative responses by children to the participants' strategies 
decreased by 64%, from 58 to 21 negative responses. Instances of participants' ignoring children declined 
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4.76 



3.31 



3.53 



2.49 



Early Childhood 
Educators 
Pre/Post 
CDR 



Middle/High School 
Educators 
Pre/Post 
CDR 
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by 90%, from 48 to 5 total instances. In contrast, among the middle and high school teachers, there was 
little change in the frequency of their interventions (2% decrease), a 28% decrease in negative responses 
from students (from 88 to 63 total negative responses), and instances of ignoring of students increased by 
5% (from 133 to 140 total instances) (Chart 5). 

For statistical analysis of these follow-up results, one-tailed dependent t tests were calculated 
using an alpha level of .05. For the early childhood participants, the results indicate statistically 
significant gains in mean skill scores from pre- to post-CDR, and statistically significant decreases in 
frequency of interventions, negative child responses, and ignoring of children. Among the middle and 
high school teachers, mean differences in skill scores from pre- to post-CDR were also statistically 
significant, while changes in frequency of interventions, negative student responses, and ignoring of 
students were negligible (Table 13). 

Chart 5. Decrease in Negative Responses From Students in a Two-Week Interval 

100 
90 
80 
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60 
50 
40 
30 
20 
10 

0 

Early Middle/High 

Childhood School 

Educators Educators 

Pre/Post CDR / Pre/Post CDR 

Magnitude-of-effect measures taking sample size into account were obtained with Friedman's 
(1968, p. 245) point-biserial correlations ( rm ) and Cohen's (1988, p. 86) power tables. Effect size criteria 
were r = .10 for small, r = .30 for medium, and r = .50 for large effects. The r m values provide 
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Table 13. Classroom Performance of 12 Early Childhood and 11 Middle/High School Educators 





Pre-CDR 


Post-CDR 


%Change 


t 


P 


rm a 


SKILL SCORES 
Early Childhood Educators 














M 


3.31 


4.67 


+41% 


2.53 


.01 


.57 


SD 


1.26 


1.81 










SE 


.48 


.34 










Range 


1. 9-6.0 


2.4-8. 1 










Middle/High School Educators 














M 


2.49 


3.53 


+42% 


1.91 


.04 


.52 


SD 


1.73 


2.86 










SE 


.52 


.86 










Range 


0-5.54 


.96-8.53 










INTERVENTION FREQUENCY b 
Early Childhood Educators 














M 


46.00 


32.43 










SD 


19.48 


13.50 


-29% 


3.00 


.005 


.64 


SE 


5.20 


3.61 










Range 


15-76 


12-61 










Total Interventions 


644 


454 










Middle/High School Educators 














M 


78.64 


76.64 . 


-2% 


.32 


.38 


.10 


SD 


43.20 


45.53 










SE 


13.02 


13.73 










Range 


8-152 


19-147 










Total Interventions 


865 


843 










NEGATIVE CHILD RESPONSE b 
Early Childhood Educators 














M 


4.14 


1.50 










SD 


5.64 


1.95 


-64% 


1.76 


.05 


.44 


SE 


1.51 


.52 










Range 


0-19 


0-6 










Total Negative Responses 


58 


21 










Middle/High School Educators 














M 


8.00 


5.73 


-28% 


.71 


.25 


.22 


SD 


18.09 


10.96 










SE 


5.45 


3.31 










Range 


0-61 


0-30 










Total Negative Responses 


.88 


63 










IGNORING OF CHILDREN 1 * 
Early Childhood Educators 














M 


3.43 


.36 










SD 


6.08 


.84 


-89% 


2.00 


.03 


.48 


SE 


1.63 


.22 










Range 


0-17 


0-3 










Total Ignoring 


48 


5 










Middle/High School Educators 














M 


12.09 


12.73 


+5% 


.15 


.40 


.05 


SD 


10.67 


9.25 










SE 


3.22 


2.79 










Range 


0-39 


0-29 










Total Ignoring 


133 


140 











a Friedman (1968) formula for magnitude of effect estimates. 
kDecrease is desired. 
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confidence that the early childhood participants' gains in skill scores and decrease in frequency of 
interventions are large ( r = >.50), whereas declines in negative child responses and ignoring are moderate 
(r = > .30). The t value for negative child response occurs at the critical point for the sample size and .05 
probability level. From a statistical viewpoint, this raises the possibility that an increase in sample size to 
about 18 participants might be needed for greater confidence in the finding regarding reduction of 
children's negative responses to the early childhood educators (Friedman, 1968, p. 247; Snyder, 2000; 
Snyder & Lawson, 1993). However, the practical significance of an actual 64% decrease in negative 
responses to the strategies during the 30-minute observations in this sample improves confidence in the 
finding. 

A similar procedure was used to estimate possible effect of the small sample size on the results 
for the middle and high school teachers. An increase of that sample size to about 14 participants might 
increase confidence in the finding that their 42% increase in skills was both practically and statistically 
significant. 

Noting that the skill scores of female middle and high school teachers increased by 71% while the 
skill scores of the male teachers increased by 17%, dependent /-tests were again used to examine 
statistical differences in their mean skill scores. No statistically significant gender differences were found 
between the group means at either the pre- or post-CDR observations, possibly a function of the very 

f 

small sample sizes (female df = 5; male df = 4). However, when mean negative responses from students 
were compared there was a statistically significant gender difference. Female teachers had a total of 5 
instances of negative student responses (mean = .8) in pre-CDR observations and 2 total instances (mean 
= .3) post-CDR, a 60% decrease. In contrast, male teachers had a total of 83 instances of negative student 
responses (mean = 16.6) in pre-CDR observations and 61 instances (mean = 12.2) for post-CDR, a 26% 
decrease. While the mean differences between the teachers in number of negative student responses were 
not at a statistically significant level (/ = 1.53, p = .08) in the pre-CDR observations, a statistically 
significant difference in the mean number of negative student responses for male and female teachers 
occurred post-CDR (/ = 2.05; p = .03). This finding suggests that the male teachers in this sample either 



O 
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had students with notably more severe behavior problems, who were less responsive to immediate 
changes in teachers' strategies. It is also possible that the strategies, designed as the basic core needed by 
all educators, were not specific to the needs of that student population with severe problems. It is possible 
also that the male teachers did not obtain full benefits from the CDR practice, even though their skills 
increased by 17% and the instances of negative responses from students to their strategies decreased by 
26%. 

Logbook records indicated that the 14 early childhood participants spent about 2 hours on average 
using the program (mean = 122.57 minutes, range = 15 - 327 minutes). There was no significant 
correlation between time they spent on the CDR and their post-CDR skill scores ( r = .33). The 1 1 middle 
and high school participants spent on average 2.5 hours with the CDR (mean = 148.27 minutes; SD = 
107.48; range = 60 - 430 minutes). Time spent with the CDR did not correlate with their post-CDR skill 
scores. 



SUPPLEMENTAL INFORMATION and UNANTICIPATED OUTCOMES 

The Georgia Department of Education indicated a high degree of interest in the PEGS! series and, in 2001 
as part of the Georgia State Improvement Grant (SIG), developed a parallel inservice course in 
collaboration with the Developmental Therapy Institute (Georgia Department of Education, 2002, 
Subactivity #10). The course, Improving Classroom Management of Students’ Disruptive Behavior, is a 
five session training program that uses the PEGS! for ELEMENTARY CD-ROM as the central content for 
10 contact hours of inservice continuing education credit (CEU). The course includes a participant's 
workbook and an instructor's guide with overheads, discussion questions, and independent assignments. 
Activities for participants to conduct in their own classrooms are also included. The course has been 
included in paraprofessional and substitute teacher training and to assist general and special education 
teachers when they team for inclusion of special needs children in regular classrooms. 
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During 2002 the course was offered through the statewide Georgia Learning Resource System 
(GLRS). Copies of the course instructor's guide and the PEGS! CD-ROM series have been distributed to 
each GLRS director in the state. The PEGS! course was taught in 7 locations by 5 GLRS projects for 73 
Georgia educators. Ratings of the PEGS! for ELEMENTARY CDR, the course, and the instructor were 
reported for 62 of these participants (Table 14). They also completed self-ratings on three dimensions of 
their own classroom behavior management, reporting (a) increased use of strategies that anticipate and 
avoid behavior problems, (b) decreased ignoring of students, and (c) increase in students' participation in 
learning activities. Additional courses are being schedule by the GLRS during 2003 and requests have 
been received for designing additional inservice courses to allow for inservice credits with the early 
childhood and secondary PEGS! programs. 

Table 14. Evaluation Reports from 62 Participants in Georgia GLRS Inservice Course Using 
PEGS! for ELEMENTARY CD-ROM 



Locations Reporting Evaluation Data 





Middle Georgia 
GLRS 
(n = 22) 


Southeast Georgia 
GLRS 
(n = 27) 


MetroWest 
GLRS 
(n = 13) 


Rating PEGS! CDR 0 


4.4 


NA 


3.6 


Rating the Course 8 


4.3 


NA 


4.1 


Rating the Instructor 8 


4.5 


NA 


4.7 


Self-Ratings of 
Classroom Carryover: 








Use of Prevention 
Strategies 15 


+14% 


+21% 


+5% 


Student Participation 15 


+20% 


+63% to +72% 


+7% 


Ignoring of Students 0 


-27% 


-20% 


NA 


8 Scale of 1 to 5, where 5 = 


highly effective to 1 = 


not effective. 





b Increase desired. 

0 Decrease desired. 
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This strong evidence of response and utilization of the PEGS! for ELEMENTARY is most 
encouraging. It is anticipated that similar trainings for the PEGS! for PRESCHOOL and PEGS! for 
SECONDARY may occur in Georgia and other states as well to maximize the benefits of this project. 



PROJECT BENEFITS 

With completion of this project, the original plan for the PEGS! three CDR-ROM series has been 
achieved. The series contributes to the improvement of instruction for all ages and grade levels. With a 
focus on promoting active student participation in learning, the series has many potential benefits. It can: 

• Meet the needs of educators to use positive behavior management strategies skillfully. 

• Provide teachers and assistant teachers with virtual learning experiences for 
independently practicing intervention strategies shown to be effective in reducing 
negative behavior of children. 

• Offer individualized feedback about the learner’s behavior management style, thereby 
assisting adults in independently self-monitoring and learning at their own rates. 

• Foster shared knowledge of developmentally and emotionally appropriate management 
strategies between general and special educators through mutual training experiences. 

• Facilitate coherent program coordination between educators and other child serving 
agencies, thereby implementing IEPs in collaborative ways. 

• Enable administrators to offer timely inservice training when new personnel or substitute 
teachers are employed, thereby addressing a need for repeated basic training in programs 
with high staff turnover. 

• Provide administrators a way to provide inservice training on an individualized basis, 
thereby efficiently using the time of supervisors and faculty. 

• Meet the needs of children with and without challenging behaviors for supportive 
learning environments in which they are willing participants. 
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Parents and Professional 
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+ First Year -H-Previous Experience 




Jielp for troubled or troubling children 

TO@tra[py ° Tte^OoSoQjg IProgtmioms 



' IDEAs' 
Work 

^ 

U.S. Office of Special 
Education Programs 



AGENDA 

Organizational Meeting of Project PEGS! 
Advisory Council 

Thursday, August 26, 1999, 4:30 p.m. 
Developmental Therapy - Teaching Programs 
Jennings Mill 

1751 Meriweather Drive, Building B, Suite 3 



Welcome and Introductions 

The Advisory Council Mission, Mary M. (Peg) Wood 
Council Chairman 

Overview of Project PEGS! Diane Wahlers, 

Project Manager 

Discussion of Ways the Council Can Respond to 
the Tasks 

Scheduling the Next Meeting 
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MINUTES OF THE ADVISORY COUNCIL MEETING 
August 26, 1999 
4:00 - 5:30 p.m. 



INTRODUCTIONS 

All members of the newly formed Advisory Council were present except Kwang Yu and Janice 
Pulliam. 

DISCUSSION OF THE WAYS THE COUNCIL CAN CONTRIBUTE 

The first task is to develop a convincing prototype for simulating problem behavior of elementary 
school age children. 

Question 1: Do you think we have identified the major categories of children’s problem 
behaviors? 

The Council review Figure 1, “Difficult or Troubling Behaviors.” A discussion followed, 
indicating that the 9 types included in Figure 1 were comprehensive, except for 3 fairly prevalent 
problems: (1) health complaints about stomach aches and head aches; and (2) bullying (a big issue in 
Georgia); and (3) blaming someone else - “It’s someone else’s fault.” These suggestions will be sent to 
the LetterPress designers. 

The discussion then centered on the role of adults in children’s behavior. It was pointed out that 
parents have few ways of getting feedback when they are doing a good job. There was also agreement 
that “95% of our problems [with children] are adults problems.” 

A second discussion focused on the pervasive blame shifting by many adults. Children have 
copied the idea that others are to blame for the consequences of one’s own behavior. It was suggested 
that some children and parents are in denial. The label “Parent Deficit Disorder” (PDD) was suggested 
as appropriate for this type of behavior in adults. It was suggested that some parents have a difficult time 
learning when to be tough (“how to be comfortable being uncomfortable”). This led to the idea that 
some parents need to learn when to be tough and when to be gentle. It was also suggested that many 
parents are doing double duty as a single parent. This burden is overwhelming - with a job requiring so 
much time and energy that it leaves little time for anything beyond basic survival skills with children at 
home. 

This discussion concluded that there might be a need for a different CD-ROM designed 
specifically for use by parents, rather than attempting to include parents as part of the user group for the 
first CD-ROM. 

Question 2: Have we left out behavior management strategies that should be included in 
Figure 2? 

The time was running out for a full discussion of this question. It will be added to the agenda for 
the next meeting. 

VIEWING THE PRELIMINARY CHARACTER DESIGNS ON THE COMPUTER 

Council members viewed 7 different character styles to provide feedback to LetterPress about 
how the visual should look. There was a recommendation to avoid clothing and hairstyles that could 
soon look out of date. There was also concern that the characters portray children in empathetic ways 
to avoid caricatures that are disparaging. There was discussion about whether it would be effective to 
actually include an adult character in the simulation. This topic will be brought up to the design people, 
and will be discussed again at the next Council meeting. 

The next meeting was scheduled for September 23 at 4:00 p.m. and the meeting was adjourned. 
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Project PECS! Prototype B Advisory Council Survey Summary, January 2000 
Project Year One, October 1, 1999 - May 2, 2000 

Features I would like to have included ... 



1. Additional supportive information in 


the introduction screen: 




Dislike 


Meet the children 


Descriptive vignettes of all the children, 
typical and special 


10096 


0 % 


Learn the interventions 


Descriptions, text and audio examples 
and non-examples of the interventions. 
Explanations of Buttons 


10096 


0% 


Tips for playing the game successfully 


How to “win” 


80% 


20% 


General Program Help 


What the game includes 


100% 


0% 


Additional background information 


a. Family and Social History 


100% 


0% 


about the children 


b. Educational History 


100% 


0% 




c. Clinical Assessment 


100% 


0% 




d. Profile of Social - Emotional 
Development 


100% 


0% 


Trial run 


Might be a down-loadable demo for 
marketing 


100% 


0% 


Additional resources 


Link to web page for more information 
about Developmental Therapy - Teaching 
and related materials 


100% 


0% 


2. Content 


Like 


Dislike 


Clarification of difference between 
spontaneous (active) participation and 
passive participation 


Slow down rate of movement for 
compliance and increase animation for 
active participation 


100% 


0% 


Children should represent relevant age 
group 


All ages 9 - 7 7; size will vary slightly 


100% 


0% 






Like 




3. Expanded Feedback 




Q 


Additional specific feedback about 
interventions chosen for a special child 


Why they did/didn’t work 


100% 


0% 


Management styles used by learner 
during game 


Anticipatory or Preventive, 
Troubleshooting, First Aid 


80% 


20% 
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Developmental Therapy-Teaching Programs 

MINUTES OF THE ADVISORY COUNCIL MEETING 

SEPTEMBER 23, 1999 
4:00 to 5:30 p.m. 

Members present. Mike Hendrick, Vicki Hunnicutt, Maggie Napoli, Janice Pulliam, Marsha Tate-Allen 

1. Update on project activities. Time lines and meetings with our new state project officer John O’Conner and LetterPress 
designers were reviewed. All projected activities are on time. LetterPress has delivered the revisions on schedule, and the design 
document for the Elementary School product has been revised, reviewed, and accepted. 

2. Review of first prototype reactions by Council members. These ideas were raised and will be brought to the attention of the 
LetterPress designers: 

** ln regard to the strategy buttons - 

♦ How can we respond when we think two strategies should be combined? 

♦ Are the strategies arranged in the order of preference? 

♦ Would it be easier to learn what the strategy buttons mean if they were arranged in order from easy to harder? 
Or would mixing them up make it a more challenging game? 

♦ Are there “good” and “not good” buttons? 

♦ The symbols were difficult to remember. Could wording be added? Or possible a practice section before 
beginning the simulation. 

** In regard to possible missing strategies - 

♦ When rules are set up and there is an infraction, I don’t see consequences. 

♦ Is there going to be a way for the adult to model respect? 

♦ Is ignoring a strategy when you make a conscious effort to ignore? Could there be an “ignore” button? 

** ln regard to the visual format - 

♦ Can all of these strategies be shown by an adult actually doing them? 

♦ Can the adult be included on the screen? 

♦ What actually ends participation? 

♦ How do you know when you won the game or did well? A “success meter” would be a good idea. Then when 
you make a good choice the child responds appropriately and the meter goes up. 

♦ Is this too stressful, to try to learn all of these strategies and use them with all the children? It’s probably 
realistic to learn to be comfortable being uncomfortable. A juggling act is realistic. 

** In regard to overall reaction to this first prototype - 

♦ The children are appealing and avoid stereotypes related to age, clothes, race, gender, culture. 

♦ Sound effects were good and help give it authentic reactions. 

♦ You could picture yourself as the teacher. 

Next meeting. The next meeting will be scheduled after the second prototype with revisions is delivered by LetterPress in January. There 
was some concern that an afternoon meeting was not possible for some working members of the Council. It was suggested that we have 
an evening meeting. 
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Developmental Therapy-Teaching Programs 
PROJECT PEGS! Advisory Council 

January 20, 2000 at 4:30 to 6:00 pm 

AGENDA 

Review minutes of the previous meeting 

Summary of problems addressed and solutions achieved at our recent 2-day work 
session with LetterPress designers 

Recommendations from Advisory Council Members 

Advisory Council members are asked to make recommendations about five current tasks: 

1 . Review descriptions of problem behaviors and rank them from most severe to least severe. This 
information will help us determine if we have provided a sufficient range in behavior problems among 
the seven children described in "Meet the Kids." 

2. Review the original "Difficult or Troubling Behaviors" list to determine if each has been included in 
the descriptions of the children with special needs. 

3. Review ethnic, racial, and gender balance among the children to be included, for both those with and 
without "special needs." 

4. Review names for the selected children both with and without "special needs." 

5 . Recommend a title for the product. 

Individual follow-up at our office 

Review the revised prototype at your leisure and give us candid feedback about your reactions and suggestions. 
Review management strategies and re- word them if needed to make them easier to understand. 

Schedule next meeting near the end of March 
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Project PECS! Prototype B Advisory Council Review Summary, January, 2000 
Project Year One, October 1, 1999 - September 30, 2000 
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PEGS! Project Advisory Council 2000-2001, Project Year 2 



Name 


Position 


Marsha Tate Allen 


Preschool Teacher, East Athens Community Center 


Geraldine Clark 


Director of Residential Services, Rockspring Homes 
Athens Housing Authority 


Carolyn Combs 


Special Education Teacher, retired 


Donna Ford 


Director, Childcare Resource and Referral Program 


Mary Hensien 


Assistant Principal, Hilsman Middle School 


Adam Kurtz 


Industrial Arts Teacher, Oglethorpe High School 


Danny McFay 


Special Education Teacher, Carson Middle School 


Drew McNeely 


Math Teacher, Clarke Central High School 


Rebecca Olson 


Director, McPhaul Child and Family Development Center, The 
University of Georgia 


Maggie Napoli 


Preschool Teacher, Emmanuel Day School and Parent of a teen with 
disabilities 


Jerry Pope 


Media Specialist, Timothy Road Elementary School 


Janice Pulliam 


Parent of teen with disabilities 
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PEGS! Project Advisory Council, 2001 - 2002, Project Year 3 
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Project PEGS (Practice in Effective Guidance Strategies) 
Advisory Council Meeting 
Monday, February 11, 2002 4:30 - 6:00 p.m. 



Introduction of Advisory Council Members 



PEGS orientation and review of progress 

PEGS for Early Childhood 

Informal demonstration 
Plan for field testing 

PEGS for Middle/High School 

Introduction to teen characters 
Ranking the Problem Behaviors 

Identification of Behavior Problems which interfere with participation in learning 



Plans for future meetings 
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Appendix B. 

Instructional Design Documents 
and 

Definitions of Strategies 
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Developmental Therapy - Teaching Programs 
with the Developmental Therapy Institute 



What to Do When...? Project 



Ckarge 

To create three animated, interactive CD ROMs for personnel and parent training in effective behavior 
management practices with children in preschool (ages 2 - 5), elementary school (ages 6-12) and 
middle/high school (ages 13 - 16). 




♦ Develop animations of children ’s troubling conduct to which adults must learn to interact in positive 

ways 

# Include instant feedback and summary of management styles 
<0 Field test, revise product; evaluate impact of product 

+ Disseminate product to Georgia public school districts and psychoeducational centers; develop 
national distribution plan 



Saype q[_ ike Project Tasks 



Design/Development 


Field Testing 


Dissemination 


Evaluation 


Content Development 
Artistic & Technical Design 
Workbook Design (SIG) 
CD-ROM Manufacture 
Workbook Printing 


In-house Testing & Revisions 
Field site selection/testing 
Observation, interview of 
selected participants 
Ongoing feedback, revisions 


Advertisement of product 
Distribution to Georgia public 
school districts and 
psychoeducational centers 
Design national distribution 
plan 


Formative evaluation 
Summative evaluation 
Statistical analysis of 
usefulness and effectiveness 



User G/myps 

• Teachers • Support Personnel * School Administrators • Parents 



Elementary School Children (SIG) 

# U.S. Dept, of Education, Office of Special 

Education 

<C> Georgia Department of Education, State 

Education Improvement Grant 
C> University of Georgia, College of Family 

and Consumer Sciences/Developmental 
Therapy-Teaching Programs Public 
Service Unit 

& Developmental Therapy Institute, Inc. 



Preschool Children and Teens (D c lE(B£SI[) 

# U.S. Dept, of Education, Office of Special 

Education 

# University of Georgia, College of Family 
and Consumer Sciences/Developmental 
Therapy-Teaching Programs Public 
Service Unit 

# Developmental Therapy Institute, Inc. 
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Project PEGS! Design Meeting Agenda 
LetterPress Software/Developmental Therapy-Teaching Programs 



Tuesday, September 21, 1999 

Present Version I Prototype 

• Discuss and refine basic product interactions 

• Discuss and refine simulation look and feel, artistic styles 

• Discuss feedback rules and requirements 

• Discuss and refine core simulation behaviors and activities 

• Discuss and refine database rule requirements (data template) 

o Discuss and refine secondary media resources (audio, video) 

• LetterPress: overview of advisory group guidelines 

• Discuss and refine product scope (number and variety of cases) and components (Presentation 
strategies, exploration strategies, practice strategies, instructional feedback options) - SIG 



Wednesday, September 22, 1999 

Discuss PEGS! design additions 

• Discuss and finalize prospective target learner audience 

• Discuss and determine critical instructional objectives 

• Discuss and draft product scope (number and variety of cases) and components (Presentation 
strategies, exploration strategies, practice strategies, instructional feedback options) - PEGS! 

• Discuss and review project communications 

• Discuss and review project responsibilities and roles 

• Discuss and review project schedule and milestones 

• Review PEGS! contract 
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Developmental Therapy - Teaching Programs/LetterPress Software 

Design Meeting, January 11-12, 2000 



Agenda 



January 11, morning 

1. Review Advisory Council suggestions for final SIG simulation prototype 

• feedback and scoring 

• management strategy summaries 

2. Finalize additional and supporting information features 

• help file: how to use the product 

• data regarding children 

• tips for being successful in both simulation and real world 

• information about Developmental Therapy -Teaching Programs 

• other useful supplemental information 

3. Review required data for State Improvement Grant 

4. Instruction on data entry and formatting 



January If, afternoon 

1. Review PEGS! Design documents 

2. Review adjusted simulation prototype (preschool level) PEGS! 

• graphical characters 

• graphical look and interface 

• feature review 

3. Discuss modifications required for teen level PEGS! 

• removing interventions 

• differences in interface 
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January 12, mowing 



1. Discuss PEGS ! additional and supporting information features 

• help file: how to use the product 

• data regarding children 

• tips for being successful in both simulation and real world 

• information about Developmental Therapy - Teaching Programs 

• other useful supplemental information 

2. Review required data for PEGS! preschool and teen versions 

3. Instruction on data entry and formatting for PEGS! 



January 72 , afternoon 

1. Review all grant requirements 

2. Review schedule for deliverables 

• LetterPress tasks 

• UGA tasks 
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Developmental Therapy -Teaching Programs / LetterPress Software, Inc. 

PECS! Design Meeting 
September 26 & 27, 2000 



Tuesday September 26, 2000 

9:00 AM: Review PEGSj for Elementary School (5/C Project) 

Up front help and documents 

Behavior meters 

Overall operation of each of the 4 scenarios 
Learner feedback 

Teacher audio for interventions / funding 1 5K? 

Web supporting documents 
Final steps to completion 

(10:15-1 0:30 Break for refreshments) 

Noon: Lunch 

1 :00 PM: Review progress of PECSj for Preschool 
Up front menu look 
Discuss up front help and documents 
Overall operation of each of the 4 scenarios 
Learner feedback 

3:00 PM: Discuss marketing ideas for_ PECS! 

4:30 PM: Dismiss 

Evening: Dinner with those available 

Wednesday^ September 27, 2000 

8:30 AM: Discuss PECS! for High School 
Interface differences? 

Graphical look of kids 
Required help and documents 
Define each of the 4 scenarios 
Learner feedback 
Time-lines for data 

Noon/1 PM: Approximate departure time 



Definitions of Behavior Management Strategies 
for the PEGS! CD-ROM Series 

Strategies That Anticipate and Avoid Problems 

POSITIVE ENCOURAGEMENT AND PRAISE. Give encouragement on positive elements in children's participation, 
not just on results. Let them know you notice something of real value in what they say and do. 

MOTIVATE WITH MATERIALS OR LESSONS. Engage children's attention and interest with attractive materials 
or academic content that has high appeal. This will create enthusiasm for getting involved and staying on task. 

ORGANIZE MATERIALS FOR CHILD (early childhood version only). Encourage independence but limit access 
to materials so that they are used in acceptable ways. Organize material for each activity so that unnecessary items are 
not distracting to participation. 

EXPLAIN PROCEDURES AND EXPECTATIONS (STRUCTURE). Provide clear expectations and then review 
what the activity or learning task involves, step-by-step. 

MODEL EXPECTED ACTIONS. Demonstrate how a task or activity will be done. Set standards for conduct towards 
others by using the same words and actions that children are expected to use. 

Strategies That Keep Behavior Problems from Escalating 

PHYSICAL PROXIMITY/ SIGNAL AWARE NESS. Communicate awareness of what a child is doing by moving 
closer, giving eye contact, or signaling with a look or smile. 

REDIRECT BEHAVIOR/REFOCUS ON TASK. Give additional guidance or re-motivate a child who shows lagging 
interest or attention. 

REFLECT POSITIVE WORDS AND ACTIONS. Comment on positive aspect of a child's words or actions (past or 
present) in nonjudgmental ways, even when problem behavior is evident. 

RULES REMINDER. Ask or remind an individual or group about the rule for expected behavior in a particular 
activity. 

INTERPRETATION (secondary version only). Connect actions to feelings by commenting on what a child may be 
feeling, based on what is seen or heard. 

Strategies That Control Highly Disruptive Behavior 

TIME-OUT IN THE ROOM. Direct child to a time-out area, with a specific objective for returning to participate. (For 
secondary student, suggest voluntary time-out in room.) 

HOLD FIRMLY (early childhood version only). Use physical means to stop a young child's highly disruptive behavior, 
by holding with firmness while communicating adult protection from losing further control. 

CONFRONT UNACCEPTABLE ACTIONS (secondary version only). Use authority to stop highly disruptive 
behavior. 

REMOVE FROM THE ROOM. Take a young child out of the room to talk about a disruptive incident and alternative 
behaviors. (For secondary student suggest voluntary decision to talk about issue). 
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BEST COPY AVAILABLE 



Appendix C. 

Text Vignette Summaries 



Continuum of Behavioral Responses Designed 
for Each Child Character 
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Summary of Student Characteristics for Early Childhood Version 
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Summary of Characteristics for Secondary Students 
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Vignettes of Child Characters in PEGS! for PRESCHOOL CD-ROM 



LE °N 

Acey is a very thin three-year-old African-American boy who lives with his single mother and several of 
her friends in a trailer at the edge of town. There are four other preschool children living there also. Acey is 
dropped off at a childcare program several days a week. He has a favorite toy bear he looks for when he 
arrives. While there, he fidgets constantly. Sometimes he wanders around the room, clutching the bear. 
Other times he stares into space. The adults at the center have never heard him speak. When they come 
close to him, he twitches, hugs the bear, and looks away. When distressed, he makes strange sounds that no 
one understands. He does the same thing when other children come close, so they ignore him. The teachers 
are very concerned. He is not interested in any activity, and does not appear to be developing like a typical 
three-year old. 

ANN 

Four-year-old Ann is a bright, outgoing African-American child who participates in group activities with 
enthusiasm and follows directions carefully. On the playground she sets the pace for the games and tells the 
others what to do. Sometimes they complain about her being "bossy". However, when another child is 
upset, Ann responds with sympathy and concern. She is among the first children to finish a task and wants 
to show her work to everyone else. If the other children do not seem interested in her work or fail to admire 
what she has done, Ann turns to the teachers for reassurance. Her favorite activity at school is Storytelling 
Time. 

BRAD 

Four-year-old Brad is a handful for his parents and teachers at the kindergarten. He tries to be the center of 
attention in every activity. Teachers report that he talks constantly and cannot sit still or stick to any 
activity for long. Impulsive and unpredictable. Brad uses obscene language and is abusive to the other 
children. Many of them are afraid of what he does and try to stay away from him. He argues with his 
teachers constantly, telling them what to do. His mother claims he inherited his "rough ways" from his Irish 
ancestors. But she is worried about his frequent attempts to strangle their cat at home. Once he tried to set 
fire to the cat. Recently his teacher found him attempting to poke a pencil down the throat of a pet rabbit in 
their classroom. The "Time Out" chair does not seem to be effective with him. When he was sent there for 
smearing paint across another child's work, he exploded violently. He knocked everything off of the 
teacher’s desk, threw the "Time Out" chair, and stormed out of the room shouting that she was a witch. 
Brad shows the most self-control and focus when he is allowed to read by himself. 

BELINDA 

Three-year old- Brenda is a shy, cautious girl who spends much of her time watching other children in her 
group. When she sees them doing something of interest to her, she will come closer to their activities and 
play by herself nearby. She prefers to play with puzzles and the toys in the homemaking area, especially the 
baby dolls and stuffed animals. Her play is imaginative and she often pretends to read to the dolls. In group 
activities, she seldom volunteers or answers adults’ questions. But when she is by herself with an adult she 
speaks quietly and spontaneously about her dolls at home. She also talks about her Grandmother’s cats and 
the good things "Grandma" cooks for her. 
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CAYLA 

Cayla is a five-year-old whose severe temper tantrums upset her teachers and the other children. When she 
cannot have her way, she screams or cries for long periods of time. Nothing seems to make her stop until an 
adult takes her outside the door where she can still see what she is missing in the room. Her mother travels 
out of town on business every week, and Cayla often says she is waiting for her mother to call her. She 
seeks approval and assistance in every activity and constantly gets up to find an adult who will admire her 
work. Any criticism upsets her and she responds by complaining that her stomach hurts or that she has a 
fever and needs to go home. When group activities begin, she insists on sitting by an adult and being first to 
have a turn. She seldom listens to directions and then pouts when the adults remind her about how she must 
do a task. On the playground she stays by herself or stands close to the adults. She refuses to eat the food 
at lunch and brings snacks from home. Cayla is most attentive and easiest to manage during Storytelling 
Time. 

CARLOS 

Carlos is a quiet five-year-old boy from a Hispanic community where English is a second language. The 
kindergarten teachers find him to be attentive but shy and unwilling to volunteer his ideas. He follows the 
rules, and responds carefully to directions from adults. His teachers are having a hard time identifying his 
real interests. They are also concerned that he often seems to be daydreaming. He hesitates to participate 
spontaneously in class but on the playground he shows leadership skills and is earning the respect of the 
other boys. His teachers hope that he will begin to make friends among the other children. At a recent 
parent-teacher conference, his teachers noticed that his ways of speaking and relating to others is quite like 
that of his father. His parents want Carlos to do well in school and are concerned that his recent readiness 
test scores show that he is behind the other children in his group — especially in reading and writing. 
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Vignettes of Child Characters in PEGS! for SECONDARY CD-ROM 



CARON 
Age 16 



Homeroom Teacher : 

Caron has average ability for academic work but isn't living up to her potential. This year she is passing 
most of her classes, as compared with less than half of them last year. Her grades could be better, but she 
doesn't want to work at it. Her attendance has improved, but she continues to be upset by the actions of 
others. She is always asking to use the restroom during class discussions. She wants teachers to do things 
for her, or to grant her special privileges. She doesn't seem to have any close friends, probably because 
she's so manipulative. She participates in class projects but quits if she can't get her way. She usually ends 
up having her way, because she annoys everyone so much with her whining and complaining. 

Guidance Counselor : 

Caron is an only child who has lived with her grandparents for the past twelve years. Her mother killed 
herself when Caron was four years old. To my knowledge, she never sees her father. Nobody likes this girl. 
She borrows others' personal things, doesn't seem to bathe often, and eats all the time. She is constantly 
whining and trying to get other students to feel sorry for her. She talks about her mother's suicide to anyone 
who will listen. She sees a therapist weekly after school. 

Caron received a psychoeducational evaluation two years ago for a possible learning disability 
when she appeared to be having problems listening, paying attention, and problem solving. However, the 
results indicated that she was not eligible for special education services at that time. The evaluation also 
indicated that while she has intellectual ability in the Low Average to Average range, her achievement in 
math calculations, math reasoning, and spelling are about three years behind that of her peers. After 
hearing these results, her grandparents got a tutor for her. 

Caron is now taking medication for depression and tells me this is why she feels so bad and has 
such erratic behavior. Last year during spring break, she overdosed on the medications, "to get high" and 
"forget my problems." I see her frequently because she drops in at my office and always seems to have 
some health concern. She complains about headaches, cramps, cold sweats, difficulty breathing, and 
numerous possible diseases. 

A Peer : 

Caron uses her background to get attention and to get teachers to help her. She told than her Dad tried to 
kill her mother. But I heard that her mother killed herself. Caron tries to get people to believe she's been 
abused and discriminated against. I think she's just a pathological liar. She tells crazy stories. Once she told 
me, "I had surgery. One of my kidneys had to be removed. I gave it to a popular cheerleader last year." 
Another time she said, she was "part royalty and related to the Duchess of Germany." 

Caron [ describing her own life to the guidance counselor]: 

It’s hard being me! I think I may have early diabetes. I can't sleep and when I lie awake, I think of why my 
mother killed herself. I try to make friends but no one is nice to me. I don't know why. I do everything I can 
to be friendly. Maybe it's because I have to live with my grandparents, and they're too old fashioned. They 
won't even let me stay home when I'm sick. And they won't let me watch my favorite TV shows very often. 
I'm thinking of going to acting school. I'd like to be on television, but they laugh at me. When I ask them for 
spending money they don't give it to me. They don't know how much clothes and make-up cost now. So the 
clothes I have to wear are gross. And the food is so terrible at school that I have to buy stuff at the store to 
eat at school. Sometimes I think I'll be just like my mother. 
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BETH 
Age 17 



Homeroom Teacher : 

Beth is hard to describe. She is so quite you hardly know she is there. She is an average student, is never a 
discipline problem, and tries hard to do what is asked of her. She has trouble in Basic Algebra and Biology 
but is doing much better in American Lit. She actually gets enthusiastic about some of the reading 
assignments. She does fairly acceptable book reviews, but is still having a hard time with spelling and 
grammar — especially punctuation. She and I had a talk recently, and she opened up a bit about how 
concerned she is about graduating. Schoolwork seems to be really hard for her. We talked about spending 
more time on assignments, but she said she stays up until after midnight every night trying to get it all done. 
She seems chronically tired to me. 

Guidance Counselor : 

Beth has an after school job that seems to be interfering with her studies. She doesn’t get off work until nine 
o'clock. Then she come home too tired to put much effort into her assignments. She says she uses her study 
hall at school to start the work, but that isn’t nearly enough time. We talked about dropping the job, but she 
says she has to work. Her single Mom doesn’t earn enough to support the two of them. Beth and I met with 
her Mom to talk about how to help. Her mother expressed concern but said, "Beth is how we keep food on 
the table." She talked about getting a better paying job herself, but says she has to go back to school to do 
this. We discussed reducing Beth's course load, recognizing that it will take Beth longer to finish high 
school. Apparently Beth’s mother wants her to graduate but thinks it is unlikely, "It’s hopeless. There’s 
nothing we can do about it." 

Other Girls : 

I don't think I've ever seen Beth enjoying herself. She's okay to be around but not very interesting. She 
hangs around our group. I guess she needs friends. But she doesn't have much to talk about. Except once 
she told us about how there was a robbery where she worked. She almost got shot. Everyone was 
impressed. She wears clothes that are a put-off. No wonder she doesn't have any boy friends. 

The Bovs : 

Beth? Beth who? Oh, that one who always has her nose in her books. Yeah, her locker is near mine, but she 
never speaks or even looks over my way. You might say she has looks, but not with those old clothes she 
wears. I don't know who she hangs out with. 

Beth [ describing her own life]: 

I never have any fun. All I do is work. It would be nice to have a close friend — someone I could talk to. I 
don’t know why I have all the bad luck. I guess I'm lucky to have a job. But the guys there make crude 
remarks to me. I’d like to slap them but I know better. I had a boyfriend there for awhile, but he started 
pushing me about sex and smokes, and that sort of thing. No thanks! 
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BB 

Age 15 



Math Teacher : 

I can't figure out BB. Last week he was in a rage and screamed at me for giving him a D. The next minute 
he apologized and calmly asked if he could discuss it with me. He wanted to do something to raise the grade 
to a C. When I told him there was no extra credit offered, he snapped into a rage again. I had to call in a 
hall monitor to escort him to the principal's office. He threatened things like, "You'll be sorry..." and "Just 
wait!" These incidents happen every week. He just can't seem to settle down. I talked with him about 
getting some help managing his anger. He smiled, thanked me, and said he was seeing a counselor and was 
trying to get his life in order. I found out two weeks later that he hadn't received or tried to find help 
anywhere. 

Principal: 

It seems every day BB is in my office with a referral from a teacher. He's always in trouble for fighting 
kids and cursing teachers. I've been trying to talk with him about this problem he has with a hot temper. He 
apologizes and says he'll change, but after he started a school-wide fight this week, I think he needs 

professional help. BB came up to one guy and started it. He said, "I think you're ugly!" The other guy 

said something back like, "You’re a jerk!" Then friends on both sides started shoving and everyone else got 
into it. 

Guidance Counselor : 

BB can't sit still when he comes into my office. He paces the floor and has a hard time staying on one 
subject One minute he's all smooth smiles and agreeable. Thai he jumps up when he doesn't like the way 
the conversation is heading. A few times he opened up to me. He told me that his parents fought all the time 
and divorced when he was sevoi. He chose to live with his father because his father told him that if he lived 
with his mother "she'll make you a sissy boy." His father thinks he is doing a good job as BB's parent. He 
says he "roughs him up to make a man of him." His father buys beer for him to teach him how to "drink 
like a man," and his friends tell BB he has "a cool Dad." But BB doesn't ever cross his Dad because if he 
did his Dad would beat him. He's learned to lie to stay out of trouble with his Dad and still get his way. 

Friend: 

He was a star on the baseball team, a great player and very popular. But one day at a game the umpire 
made a call he didn’t like, and BB threw the bat, walked over to the guy and started to scream at him. The 
guy yelled back, and BB picked up the bat and hit him across the face. It broke his jaw. BB was kicked off 
the team, and our team has been losing ever since. But BB's a cool guy to hang out with. He's one of my 
best friends, but I don’t cross his path! Most people are terrified of him, especially 'cause he's smart. 

Peer £a friend of BB's girl friend. Celeste]: 

BB is very popular — with lots of friends. I guess that's why Celeste falls for him. She's always telling me 
about the romantic things he gave her when they first started seeing each other. I was jealous, but that 
faded when Celeste came over to my house with black and green bruises on her face, saying that she and 
BB had gotten into a little fight He’s real persuasive with her until she gives in to him. Now he thinks he 
owns her, but Celeste doesn't see it, "He can be so sweet!" 

BB [d escribing his own life to the principal]: 

I visit my mother every now and then, but I don't stay long. She has a boy friend that's a jerk. He likes to 
pick fights with me. He makes sarcastic remarks like, "Did you beat up anybody today?" So, I have to let 
him know he’s going to get it, too, if he doesn’t watch his mouth. Then he badmouths my Dad with dumb 
remarks like, "You're just like your old man — not worth much except to fight." I'm not letting him get 
away with that! 
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ALEESA 
Age 15 



Homeroom Teacher : 

Aleesa is a very good, straight A student and a great one to have in class. She's always willing to help. If 
there is a group project, she's reliable to see that everyone gets involved, and the job gets done. She was 
selected to be a JV cheerleader. This is a big honor. The school has a service club which she is in. They 
visit the elderly two Saturday’s a month. She's also in student government and secretary of her class. Aleesa 
has lots of friends from diverse backgrounds. That's one reason she's a leader here. 

Guidance Counselor : 

I don't see Aleesa often, except when we have student government meetings. She knows how to be a good 
member. She doesn't have a lot to say, but when she speaks up, she usually gets the attention of the other 
Council members. I think she is a very positive influence for the standards of conduct we try to maintain in 
this school. I remember one time when she was quite outspoken about some of the bullying that was going 
on. I think our student government had a lot to do with turning that problem around. Now there are very 
few problem kids still around trying to dump their own feelings on others. 

A Close Friend: 

When I first met Aleesa I just wanted to hang out with her because she always had a lot of guys hanging 
around her. But then I found out that she is a really nice person, and we've been friends ever since. It 
doesn't matter how pretty she is; it's her personality. She never says anything mean about anybody, and 
she's nice to people of every race. 

Other Girls : 

Aleesa's a lot of fun. We go to the movies together and spend-the-night at each others' houses on the 
weekends. She is there for you if you get stressed out, and she can always see a good side to things, no 
matter how awful. 

The Bovs : 

Some girls act like they can't stand you. Aleesa's not one of them. She always seems interested and she's 
easy to talk to. Sometimes she is too straight, but mostly, she keeps her ideas to herself. But if she doesn't 
like what someone is doing, or disagrees with what they've said, she lets you know it — without making 
you feel stupid. 

Aleesa f describing her own life]: 

My parents divorced when I was in fourth grade, but they are on friendly terms. I see my Dad on weekends, 
but that sometimes gets unpleasant. He has his own ideas about dress code and curfew hours. He doesn't 
understand that there has to be some give and take in high school. He has a lady friend who he wants me to 
like. She's not the type I would choose, but he's the one who has to live with her. My Mom seems to 
understand me better. She gives good advice and encourages me. Mom also sticks to her rules, which I 
don't mind. When we have arguments, she'll listen, but she also tells it like she sees it. She's sort of like a 
sister. I wish I had a brother and sister. I think both of my parents are proud of me, and expect me to go to 
college. I dont know how they can afford it. Maybe I can get a scholarship. 
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JAMEEL 
Age 17 



Homeroom Teacher : 

Jameel has never been a problem in class but his lack of interest in his schoolwork and his lack of friends 
has started to worry me. When he comes to school, which is infrequently, he just sits and daydreams. Or 
maybe he's just asleep with his eyes open. He's failing all his classes this quarter and says he just doesn’t 
care. He also talks about feeling useless and has mentioned suicide. He has shown his classmates scars on 
his wrist and says, "Nobody would care if I'm dead anyway." I suggested that he see the guidance counselor 
to sort through his problems. 

Guidance Counselor : 

Jameel has a history of low achievement and absenteeism from school. Last year a psychoeducational 
evaluation concluded that Jameel met eligibility criteria for special education services through the 
emotional/behavioral disordered program. The evaluation summary also indicated that he should be 
referred for mental health and psychiatric services. However, his mother never agreed to these 
recommendations. When I try to talk with Jameel, he is somewhat secretive, but he seems to want 
others to know how bad things are for him. Jameel is aware of the difficulties he is having at school and 
describes the classes as "boring, too crowded, and noisy." He also feels that students are talking about him 
in the cafeteria and halls. Jameel has told other students that he has a police record. And he talks to me 
about his two arrests last year for buying drugs from a dealer who hangs out in his neighborhood. 
Apparently he is trying to escape from his situation by solitary drinking and drug use. I've tried to contact 
his mother but can't reach her. 

A Peer: 

Jameel has drifted down to the hard core drug users at school. Everyone else is scared off by his weirdness. 
He's really weird. He never mentions doing anything like school parties, or sports, or friends. He always 
talks about being worthless, and he has shown me scars he says he did to himself. Once you come up along 
side him and start talking he’s very responsive. I told him once, "I’m on your side. You've been through 
hell." He told me then that the only way he has fun is to cut himself. He probably does it for attention. 

Jameel f describing his own life to the guidance counselor]:. 

Yeah, I know I'm not going to graduate. I have bad dreams. Sometimes my hands sweat a lot. My Dad's in 
jail. My Mom’s at work all the time. I just do my own thing and nobody bothers with me. If I'm not home 
when she gets back from work, she doesn’t worry. She figures I'll get back sometime ... She's so burned 
out. Once I got hired to help make hamburgers part-time. I got some spare cash, but it didn't work out." 
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VICTOR 
Age 16 



Homeroom Teacher : 

Sometimes I worry about Victor. He is a new student and so reserved that I can't tell whether he is really 
understanding the material or not. I don’t think his low grades reflect what he really knows. He certainly 
tries in class. He completes assignments, although not always correctly, is attentive, and is always polite. 

He seems to hang back from getting involved with the others, but responds to the students when they 
initiate exchanges with him. I noticed that in P. E., the boys seem to have accepted him and seem eager to 
have him on their team. I am concerned that Victor is falling behind in his studies. When I talked with the 
guidance counselor about him, she suggested that we have a parent conference to review the files from his 
previous school and plan ways to help Victor stay up with the class. 

Guidance Counselor : 

The standardized achievement test scores from Victor's previous schools indicate that he has steadily fallen 
behind a little each year since third grade. His test scores show that he has low-average ability and should 
be able to do the academic work. But it may take some additional tutoring. Our school has a mentoring 
program that might be able to provide this help. It may be that Victor also has some problems with English, 
because Spanish is the only language spoken at home. His parents have agreed to Victor’s meeting with the 
teacher who has classes in English as a Second Language. We will know more about how to help Victor 
after that session with her. 

A Peer: 

Victor's okay. He doesn't say much but that's not all bad. There are plenty of other guys who have too 
much to say. He knows how to stay out of trouble. Maybe girls like quiet types like Victor. He may be 
lucky that way. 

Victor [ to the guidance counselor]:. 

School's okay. There are some great guys here. The girls are pretty nice, too — most of them. Home? Well, 
they don't speak much English. They want me to be a good student. My mother and my father, they neither 
one went to school much. But they can read good in Spanish. They want me to go to college. That's a 
dream! I want to get a job. 
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Formative Evaluation of 
PEGS! for Preschool 
by College Students 



Georgia College & State University 
November 20, 2000 



l*E€»S! Field Testing: Focus Group Questions 



General Content 


1 . What is the purpose of l*EI5S! for Elementary School Children? 


Technical Difficulty 


2. What are your comments about using this program? 


Instructional Effectiveness 


3. What did you learn from your experience with the program? 


Instructional Difficulty 


4. a. What starting level did you choose? Was it right for your? 

b. What is the most appropriate, beneficial practice level for you now? 

c. What activities did you select? 


Instructional Relevance 


5. a. How relevant are the child characters to real-life children? 
b. How typical are the activities which are presented? 


Identifying problem 
behaviors/applying 

effective strategies 


6. a. Did you read/use the children's files? 

b. Were the files helpful in understanding the needs of the children? 

c. How did you learn to match the needs of the children with effective 

strategies. 


Feedback 


7. a. How beneficial was the feedback you got during the simulations? 
b. What did you learn about yourself from management style feedback 
at the end? 


General 


8. Who might benefit from this CD? 


Supplementary Information 


9. a. What background, supplementary, or follow-up material would be 
helpful to you (or others)? 
b. What changes would you recommend? 
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Project PEGS! Focus Group Characteristics 
Project Year Two, May 3, 2000 - February 7, 2001 

Focus Group Characteristics 



Total Participants 


n = 16 


Age 


88% 20 - 30 Years 


12% 30 - 40 Years 


Gender 


12% Male 


88% Female 


Ethnic Background 


69% European American 
6% African American 
6% Hispanic American 
6% Native American 
12% No Response 


Occupation 


88% Students 


12% No Response 


Years of Professional Experience 
With Children 


12% <1 Year 
12 % 2-3 Years 
25% ^5 Years 


44% None 
6% No Response 


Ages of Children Worked With 


25% 5 Years and Younger 
44% 6 -12 Years 
12% >12 Years 
3 1% No Response 


Number of Children Worked 
With Daily 


25% 6-10 Children 
25% 11-20 Children 


19% > 20 Children 
3 1 % No Response 


Number of Children Worked 
With In A Typical Small Group 


63% ^5 Children 
31% 6-12 Children 


3 1 % No Response 


Family Experience 


25% Are Parents 

(3 of 4 parents have children 

age 5 and older) 


75% Are Not Parents 


Computer Experience 


6% Not Very Experienced 
69% Moderately Experienced 


19% Very Experienced 
6% No Response 
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2. Mastering the”Came:” Interacting with the Program 
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4. Introductory Content: Menu Screen 
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Appendix E. 

Formative Evaluation of PEGS ! for Middle/High School 
by College Students 
An Evaluation Report by 
Emise S. Singleton 



PEGS! for Middle/High School 
Practice in Effective Guidance Strategies 



An Evaluation Report 
Prepared by 



Emise S. Singleton 
Doctoral Student 
University of Georgia 



November 25, 2002 



Executive Summary 
Introduction 

This evaluation was conducted to determine the usefulness of PEGS! for Educators in 
Middle/High Schools-(Practice in Effective Guidance Strategies). PEGS! is an interactive 
CD-ROM for adults who work with students in middle and high schools. The interactions that 
occur on PEGS! allow the user to learn strategies to use in the real life classrooms when faced 
with various behavior management challenges. 

PEGS! was evaluated using during October of 2002 using three methods: 

1 . Checklist-completed by a sample of students who will work in the educational setting in 
the future 

2. User interview-completed by instructional design students 

3. Expert review-completed by instructional design students 

Overview of Results 

Overall the participants felt that the content of PEGS! was practical and effective. The behaviors, 
actions, and cultural diversity are representative of a typical middle/high school setting. The 
results also indicate that the CD-ROM could be a valuable learning tool for educators. Most of 
the participants in this evaluation agreed that using the CD-ROM was not difficult; however, 
some of them were unsure of where to begin upon entering the program. 

Overview of Recommendations 

By and large the participants in the study were impressed with PEGS! The participants found that 
the CD-ROM would be useful for teachers in middle/high school with a few changes. The 
recommended changes are mainly cosmetic and will afford the user the opportunity to work 
through the CD-ROM with more ease. 

• The CD-ROM should provide explicit directions upon beginning the program. A splash 
screen or directions written in bold print at the bottom of the opening screen would help in 
guiding the user. 

• Users would be aided by the addition of names to the icons in the activity center. When 
users place the mouse over the icon a box would appear indicating the name of the 
strategy. 

• The icons used in the student activity center (the strategies) should more clearly match 
what they appear to mean. For example, the "ear icon" means "connect actions to 
feelings". Most users would associate an "ear" with listening. 

Once the above recommendations are implemented, PEGS! will serve as an effective tool for the 
beginning teacher, paraprofessional or parent. 
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PEGS! for Teachers in Middle/High Schoob Student Questionnaire 
These questions are focused on your reactions to the PEGS! CD-ROM. 

Circle the number that most closely corresponds to your opinion about each statement 
1 = Disagree strongly; 2 — Disagree somewhat; 3 = Neutral; 4 = Agree; 5 = Agree Strongly 

About The Content 



The graphics represent a typical middle/highschool. 

Activities portray typical classroom situations. 

The students represent different personalities that might be found in a 
typical school. 

The students are culturally diverse. 

Behavior problems presented are typical for middle/high school students. 
Audio adds to the authenticity of the situations. 

Students’ remarks are sufficiently typical for the presented situations. 
Teachers’ remarks are sufficiently typical for the presented situations. 

Aboat the CD-ROM as a Learning Opportunity 

Information about the students is presented in a culturally sensitive, 
professional matter. 

The menu offers easy access into the simulations. 

The presentation format engages me. 

The program held my attention. 

This is a useful way to practice managing behavior problems 
of middle/high school students. 

About Using the CD-ROM 

Operating the program is not difficult. 

It can be used with very little computer experience. 

It is easy to change from one activity to another in the program. 

Stopping and reentering is not difficult. 

Feedback is quick and to the point. 

I used this program without assistance. 

I think this CD-ROM could be useful for teachers in 
middle/high schools. 

Care to Comment Farther? (Use the back if needed) 



2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 



4 

4 

4 

4 

4 

4 

4 

4 



4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 



5 

5 

5 

5 

5 

5 

5 

5 



5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 



4 5 



How many times did you play or replay this CD-ROM? In total, how many minutes did you spend with this CD-ROM? 




E-3 

* * 9 best copy available 



o> 

P 6 

s 

CZ> 

a 

a> 



C/3 



O 

o 

(A 

V 

OX 

ca 

u 

« 

> 

◄1 



>> 

OJ MM 

3 OX 
u a 
ox o 
< i 



a> 

a> 

u 

OX 



a> 



« a 

g -S 

OX £ 

S 1 
5 o 



a» X* 
£- OX 
ox a 

C 3 Q 



V) 



C/5 



o 

ERIC 



& 






ss 



ox 

a 

*e 

« 



^8 



*5 



ox 

C 

e 

cd 



2 

0 

« 

1 

Q 

U 

<U 

J 3 

oxj 

G 



I 



£ 



M 



2 



2 

■ 

O 

V 

<u 

JS 

+* 

on 

a 



J 3 

ox 

I 

-a 

T 3 

s 

c 

E 

XJ 



<2 



<u 

-O 



3 

s 

s 

o 

§ 

<j 



J3 

♦* C^* 

>V 5» 

G O 

55 

^ M 



100 



BEST COPY AVAILABLE 



Average Time Spent 
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User Interview Protocol 



Reviewer: Due Date: 

The following questions are based on your reactions to the PEGS! CD-ROM. PEGS! is an interactive CD- 
ROM for adults who work in the middle/high school setting. The objective of PEGS! is to provide adults with 
positive training in effective behavior management techniques. 

1 . Please describe your use of PEGS! 

I went to each of the different activity places and explored each one. 

2. Please describe your first reactions to PEGS! 

Kind of cool. I never thought that this kind of training, to train teachers to react to students, the 
counseling training could be made on a CE-ROM. I think this is a good idea. 

3. Please describe your present opinions of PEGS! 

Mmmmmm. . . sometimes I’m not real patient to read the words so I would like it too be more active, more 

animation. 

4. Do you need additional training in order to utilize PEGS! 

No, I don ’t think so, I think you need very little guidance in order to do the program. 

5. To what degree did you accomplish the performance objectives established for PEGS! 

I’m not a teacher so I’m not sure if I achieved what I was supposed to. I found that the students 
behaviors occurred extremely fast and I couldn ’t always react to them. 

6. What would you tell another person about to use PEGS! for the first time? 

Be patient, a lot of stuff on the CD you have to be patient to run through it. 

7. What kinds of successes have you experienced with PEGS!! 

8. What kinds of problems have you experienced with PEGS! 

I can ’t react to the students reaction so quickly. I need it to slow down. 

9. Please describe the areas in which you feel most competent concerning use of the PEGS! 

1 0. Please describe the areas in which you feel lease competent concerning use of the PEGS! 

Reacting to the student ’s behaviors. It was too fast for me. 

E-6 
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1 1 . What improvements would you recommend for PEGS! 



add labels to the icons 



more animation 

more difference between the three levels 
some hints after I practice 

I think the CD should run automatically when you place it in the CD drive 
too many words, not enough visuals.. .like in the resource center 

the icons under the strategies should be clickable, I may only want read about one of the 
strategies, currently I have to select the major category and then read down, / should be able 
only select the strategy I want to know more about. 



User Interview Protocol 



Reviewer:— Due Date: 

The following questions are based on your reactions to the PEGS! CD-ROM. PEGS! is an interactive CD- 
ROM for adults who work in the middle/high school setting. The objective of PEGS! is to provide adults with 
positive training in effective behavior management techniques. 

1 . Please describe your use of PEGS! 

Initially I had some problems not know how to go about clicking on the icons because there was 
in introduction as much as I would like it to have. The beginning has a small pop-up dialog box 
and I would like for it to explain more so what to do. I realize that it would be good if you go 
from left to right. 1 went to the guidance office first and wasn 't sure what to do. Once 1 went back 
to the main screen and went to the teacher ’s workroom it will give you the objectives and how to 
play. So 1 was thinking that “How to Play ’’ should be first, that maybe this Memorandum 
should be the first thing to come up when the program opens. It has all of the explanations, how 
to use. 

2. Please describe your first reactions to PEGS! 

I really like it. It gives you scenarios of how to react, manage students behavior. Maybe there 
should be slightly more behavior characteristics because after a while you just click the same 
thing for the same type of behavior. 

3. Please describe your present opinions of PEGS! 

/ think this program will not be as effective for teachers in the classroom. I think it will be very 
helpful for the preservice teachers because they have not had that experience yet. It will prepare 
them to manage those various behaviors. 

4. Do you need additional training in order to utilize PEGS! 

No, I don ’t think you need any additional training if the “memorandum " appeared when you 
first start the CD. Maybe a recommendation telling them to go to Teachers ' Workroom, 

Resource Center, followed by the Guidance... I would like the sequence to be presented. The 
Activity Center will be last because you practice classroom management there. 

5. To what degree did you accomplish the performance objectives established for PEGS! 

Most of the time 1 used positive reinforcement kind of comments. The thumbs up, smiley face, 
most of that time and that pretty much worked. I didn ’t play the advanced mode, 1 played 
beginner and intermediate, maybe in the advanced mode you would have outburst, violent 
outburst. I didn ’t play that one but 1 should have. 

6. What would you tell another person about to use PEGS! for the first time? 

It is an interesting and helpful CD especially if you are a newcomer to classroom management. It 
is something I would like to have if lama beginner, a newbie to teaching. I would appreciate 
such software. 
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7. What kinds of successes have you experienced with PEGS! 

I am was able to manage a class and I think their mood of the class improved and teaching is 
more effective. 

8. What kinds of problems have you experienced with PEGS! 

Using it initially, knowing what to do at the beginning. 

9. Please describe the areas in which you feel most competent concerning use of the PEGS! 

The static part, because it is not, you don 7 have the time constraints, you can take your own time 
browsing and looking at it. When I came to the student activity center if you don 7 do anything 
you know the class starts to act up. 

1 0. Please describe the areas in which you feel least competent concerning use of the PEGS! 

The student activity center became there isn 7 a set solution. I think that is the most interesting 

part. 

1 1 . What improvements would you recommend for PEGS! 

On the opening screen the instructions at the bottom could be more prominent, brighter... bolder. 
Other than moving the directions to the first screen that is it. The guidance office jumps out at 
me so I went there so I clicked and the student activity center stands out so 1 went there became 
it is more prominent. So I went there and 1 tried to tinker around and 1 realize 1 should have 
come here first. 
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Expert Review Form 



Reviewer: Due Date: 

The following questions are based on your reactions to the PEGS! CD-ROM- PEGS! is an interactive CD- 
ROM for adults who work in the middle/high school setting. The objective of PEGS! is to provide adults with 
positive training in effective behavior management techniques. 

Please circle your rating and write comments on each aspect of the interactive multimedia package. 1 
represents the lowest and most negative impression on the scale, 3 represents an adequate impression, and 5 
represents the highest and most positive impression. Choose N/A if the item is not appropriate or not 
applicable to this CD-ROM. Use back of this paper or additional paper for additional comments. 

NA^Not applicable l=*Strongty disagree 2= s Disagree 3»Neltber agree/nor disagree 4= Agree 5= s Strongly agree 



AREA 1 - INSTRUCTIONAL DESIGN REVIEW 



AVG. 

Score 



1 . PEGS! provides learners with a clear knowledge of the 
program objectives. 

2. The instructional interactions in PEGS! are appropriate 
for the objectives. 

3. The instructional design of PEGS! is based on sound 
learning theory and principals. 

4. The feedback of PEGS! is clear. 



N/A 1 



N/A 1 



N/A 1 



N/A 1 



3.5 



5. The pace of PEGS! is appropriate. 



N/A 



2 3 4 5 



3 



6. The difficulty level of PEGS! is appropriate. 



N/A 



2 3 4 5 



3.5 



COMMENTS 

• “Memo random” is misspelled, should be “Memorandum” 

• on the Tips screen it refers to “these 5 tools” when only 4 are listed 

• I’m not sure if the final product will run automatically when placed in the CD-ROM drive, if not, include 
instructions 

• When the program first loads, it would be better to add an element that 

1. explains what is about to happen 

2. allows the user to skip the intro music. I can actually skip the intro by clicking anywhere on the screen, 
but there is no indication of this. Two buttons “Overview” & “Click to Enter” would be appropriate. 

• The interactions do seem appropriate for the objectives, but the design needs a little work on the details of 
the presentation. I found the strategy icons difficult to remember and when I selected one, the delay between 
selection and hearing the response it represented was significant and made it difficult that while I was 
waiting to see if that worked or not, that another student would present a problem and there was nothing I 
could do about that one while I was working on the first one. 

• I found immediate feedback from strategy icons slow and confusing. It was repetitive and didn’t allow me to 
keep track of what had worked. It would be useful to get some kind of running tally and to have suggestions 
of why or why not a strategy succeeded or failed with a particular type of student and have suggestions about 
how to improve something in the future. 
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AREA 2 - COSMETIC DESIGN REVIEW 

7. The screen design of PEGS! follows sound principles 


N/A 


1 


2 


3 


4 


5 


AVG. 

Score 

4 


8. Color is appropriately used in PEGS! 


N/A 


1 


2 


3 


4 


5 


5 


9. The screen displays are easy to understand. 


N/A 


1 


2 


3 


4 


5 


3.5 



COMMENTS 

• The Sound made when the menus are closed is a little jarring. 

• The scrolling of the credits takes a long time, if I were looking for something at the end Pd rather navigate 
through a few credits pages. 

• It took me a little while to understand what I was supposed to do. It might be good to have some kind of 
splash page or something that gave a brief description of what the purpose of the site was and how to use it 

• I liked the colors and the graphics and fonts were appealing. 

• The screed displays could use some improvement, in particular the strategy icons. I couldn’t figure out why 
an ear stood for “connect actions to feelings” for example. It seemed like it should be something to do with 
listening to the student 



AREA 3 - PROGRAM FUNCTIONALITY REVIEW 



10. PEGS! operated flawlessly. 



N/A 



AVG. 

Score 



COMMENTS 

• Levels - only mention of the beginning, intermediate, & advanced levels that I could find was in the How to 

play section, which really only mentioned the first two. I would suggest describing the two levels better. How 
do I know which level I am, and what 




1 0 7 



E-l 1 



Appendix F. 

Georgia Plan Concerning 
Parallel Distribution of the Prototype Version 
for Educators in Elementary School 
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Georgia Department of Education 
Office of the State Superintendent of Schools 
Twin Towers East 
Atlanta, Georgia 30334-5001 

WebPage: http://www.doe.kl2.ga.us 

Linda C. Schrenko 

State Superintendent of Schools (404) 656-3963 — FAX (404)-65 1-6457 

February 15, 2001 



Mary M. Wood, Ed. D. 

Developmental Therapy - Teaching Programs 
P.O. Box 5153 

Athens, Georgia 30604-5153 
Dear Dr. Wood: 

The PEGS! in Elementary School CD-ROM from Developmental Therapy - Teaching Programs is a valuable 
teacher training tool to improve appropriate behaviors in students. You and your staff have worked tirelessly to 
develop a valid, comprehensive tool that will increase student outcomes. 

Over the last month, we have introduced the CD-ROM to educational leaders across Georgia. They found the 
CD-ROM to be an engaging and promising way to build teachers' skills. They were especially impressed by the 
results of the pilot study of CD-ROM users that showed that teachers increased their use of appropriate 
strategies by over 20% which resulted in a reduction of negative student behaviors by 58%. In March 2001 , we 
will distribute the CD-ROMs to the 1 7 Georgia Learning Resources System centers to use with service 
providers across the state. The CD-ROM will be available to service providers in two contexts: as a stand alone 
instructional tool and embedded in a course that you are currently developing. 

Many inquiries have been made about your forthcoming programs, PEGS! in Preschool, Middle School, and 
High School. This innovative series of three CD-ROMs promises to have a positive impact on teachers and 
students in school environments. 




Georgia Department of Education 



JO:lah 
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Appendix G. 

Rating Scales 
for the 
CD-ROMS 



PEGS! FOR TEACHERS IN PRESCHOOL: FIELD TRIAL EVALUATION 
Circle the number that most cllosely corresponds to your opinion about each statement 
(5= Agree strongly; 4= Agree; 3=Neatral; ^Disagree somewhat; ^Disagree strongly.) 



ID# 



About the Content 

The PEGSl children have behaviors that I have seen in real children. 

Their difficult behaviors are real-life challenges. 

The activities are typical of early childhood settings. 

The strategies are ones I have heard of. 

The child characters responded to the interventions in real-life ways. 

The responses of the child characters were appropriate for ages and individual needs. 

About the CB as a Learning Opportunity 
The child characters held my attention. 

Some behaviors were easy to guide to participation. 

Other behaviors were challenging to guide to participation. 

The program choices allowed for my preferences. 

There was enough information about the children. 

Information about the children was presented in a professional way. 

The information about the strategies was helpful to me. 

The strategies are practical in real-life situations. 

Feedback at the end provided me with useful information. 

The suggestions for improving my management style were helpful. 

About Using the C]D) 

It can be used with very little computer experience. 

Operating the program presents no problems. 

The format is easy to follow. 

Information about how to play was clear. 

It is easy to change from one activity to another in the program. 

Stopping and reentering later is not difficult. 

Feedback is quick and to the point. 

I used this program without assistance. 



5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 



4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 



3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 

3 2 



Please write any comments you may have on the back of this sheet . THANKS FOR HELPING ! 
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PEGS! FOR TEACHERS IN MIDDLE/HIGH SCHOOL: FIELD TRIAL EVALUATION 
Circle the n timber that most closely corresponds to your opinion about each statement 
(5= Agree strongly; 4= Agree; 3=Neutral; 2=Disagree somewhat; l=Disagree strongly.) 



About Using the PEGS? CD-ROM 
Operating the program is not difficult. 

It can be used with very little computer experience. 

The format is easy to follow. 

Information about how to play is clear. 

It is easy to change from one activity to another in the program. 

Stopping and reentering later is not difficult. 

Feedback is quick and to the point 
I used this program without assistance. 

About the PEGS! CIP-ROM 

Tlie purpose of the program is to practice guiding students to participate. 

The game format makes the practice fun. 

The program choices allow for preferences among difficulty levels. 

There is sufficient information about the students. 

Information about the students is presented in a culturally sensitive, professional way. 
Some student behaviors are easy to guide to participation. 

Other behaviors are challenging to guide to participation. 

The information about the students’ problem behaviors is useful. 

The strategies are practical in multicultural, real-life situations. 

Feedback about each student provides needed information. 

At the end of the program, suggestions for improving management styles are helpful. 

About the Content of the PEGS! CID^ROM 

The students have behaviors that I have seen in real students. 

Their difficult behaviors are similar to real-lief challenges teachers may face. 

The activities are typical of those in real school settings. 

The strategies are ones middle/high school teachers should know about. 

The students’ responses to the strategies are typical of real-life students. 

About the Usefulness of the C0-ROM 

I recommend this PEGS! CD-ROM to others working in positions similar to my own. 



5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 



Please ivrite any comments you may 



have on the back of this sheet . 



4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 



3 

3 

3 

3 

3 

3 

3 

3 



2 

2 

2 

2 

2 

2 

2 

2 



3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 



2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



2 

2 

2 

2 

2 
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Appendix H. 

Rating Adult Strategies: 
Early Childhood Educators 
Middle/High School Teachers 



These instruments are abbreviated versions of the Developmental Therapy - Teaching Inventory 
of Teacher Skills (DTRITS), (Wood, 1996, Appendix 2, pp. 328-332). 
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Adult code number:. 



_ Pre Date:- 
Post Date:. 



Activity:. 

Activity:. 



Number students:. 
Number students:. 



RATING ADULT STRATEGIES: EARLY CHILDHOOD EDUCATORS 



Directions: This list of basic adult strategies includes descriptors proven effective in increasing student participation in planned 
educational activities. As you observe, keep a tally of the number of occurances of each strategy during the specified time period. 
Then, review the descriptors within each strategy category and rate the quality of use of each item. Keep in mind that “effectiveness” 
is defined as positive participation by the students in the activity. 

Circle PP if an item was used in a positive way by the teacher and effectively improved child participation. 

Circle NP if the strategy was used in a negative way by the teacher, but effectively improved child participation. 

Circle PN if the strategy was used in a positive way by the teacher, but did NOT effectively improve child participation. 
Circle NN if the strategy was used in a negative way by the teacher and did NOT effectively improved child participation. 
Circle NO if the strategy was not observed. 

Circle UN if the strategy was observed but unnecessary to improve child participation. 

Circle IN if the strategy was used inconsistently. 

Ignore Child’s Behavior 



Adult appropriately ignores a child when the 
child is participating in the task. 

Adult appropriately ignores a child’s behavior when the 

child is merely "testing" adult to elicit a reaction. 

Adult does not ignore a child’s behavior when child 
is in crisis or in need of adult assistance. 



PP 


NP 


PN 


NN 


NO 


UN 


IN 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


PP 


NP 


PN 


NN 


NO 


UN 


IN 



Encourage and Praise 



Interpersonal forms of praise and encouragement 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


are used frequently with all children. 
















Adult uses praise and positive statements rather 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


than negative statements. 














IN 


Adult conveys personal recognition of every child. 


PP 


NP 


PN 


NN 


NO 


UN 


Each child is frequently contacted by the adult 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


in supportive ways. 












UN 


IN 


Small accomplishments are recognized. 


PP 


NP 


PN 


NN 


NO 


Adult conveys personal recognition of child 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


as important individuals. 
















Use Motivating Materials and Activities 












Activities promote pleasure and participation. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


from each child. 
















Activities are used to stimulate individual skills. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Activities do not extend beyond peak of motivation. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


(Circle PP if task does not extend beyond peak.) 














IN 


Activities provide opportunities for using newly learned skills 


PP 


NP 


PN 


NN 


NO 


UN 


independently. 












UN 


IN 


Materials are used for specific purposes which encourage participation 


PP 


NP 


PN 


NN 


NO 



in the learning task. 
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Circle PP if an item was used in a positive way by the teacher and effectively improved child participation. 

Circle NP if the strategy was used in a negative way by the teacher, but effectively improved child participation. 

Circle PN if the strategy was used in a positive way by the teacher, but did NOT effectively improve child participation. 
Circle NN if the strategy was used in a negative way by the teacher and did NOT effectively improved child participation. 
Circle NO if the strategy was not observed. 

Circle UN if the strategy was observed but unnecessary to improve child participation. 

Circle IN if the strategy was used inconsistently. 

Organize Materials for Child 



Adult assists child in control of materials 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Materials provide for successful participation. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Adult encourages child to use materials independently. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Specific areas of the room are designated for certain activities PP 


NP 


PN 


NN . 


NO 


UN 


IN 


Describe Procedures and Provide Clear Expectations (Structure) 








Activities requiring physical movement are interspersed with 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


those which are less active. 
















Before activity begins, adult “talks through’Vdemonstrates 


PP 


NP 


PN 


NN 


NO 


UN 


IN' 


the activity, when necessary. 
















Children know the behaviors expected in each activity. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Schedule of activities is posted in room. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Expectations are clear and meaningful to children. 


PP 


NP 


PN 


NN 


NO 


UN 


■ IN 




Model Expected Actions 














Adult demonstrates or “talks through” activity to 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


provide model for participation. 
















Adult demonstrates expected responses. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Adult provides positive interpersonal model. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Move Closer to Child (Proximity) 












Adult moves near child as needed. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Adult uses touch to convey guidance or support. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 




Redirect Child to Activity 














Adult uses verbal redirection, statements, or questions 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


to help children participate. 
















Adult responds to a child’s inappropriate comments 


PP 


NP 


PN . 


NN 


NO 


UN 


IN 


with a question or refocuses child on task. 
















Adult uses physical touch to redirect a child to participate. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Reflect Child’s Positive Words or Actions 













Adult puts children’s experiences into words. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Adult finds some positive aspect in a child 
who is misbehaving. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Adult reflects what children are feeling, to 
convey understanding. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 
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Circle PP if an item was used in a positive way by the teacher and effectively improved child participation. 

Circle NP if the strategy was used in a negative way by the teacher, but effectively improved child participation. 

Circle PN if the strategy was used in a positive way by the teacher, but did NOT effectively improve child participation. 
Circle NN if the strategy was used in a negative way by the teacher and did NOT effectively improved child participation. 
Circle NO if the strategy was not observed. 

Circle UN if the strategy was observed but unnecessary to improve child participation. 

Circle IN if the strategy was used inconsistently. 



Remind Child of Rules 



Rules are “meaningful” and developmental^ appropriate. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Classroom rules are few and stated positively. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Classroom rules focus on helping child be successful. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Consequences for breaking rules are stated constructively. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Remove from the Group (Time-out) 

t 












Time-out from the group, in room, is used if needed. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Child understands why he/she was sent to time-out. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


The interpersonal exchange between child and adult is positive. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Time-out is brief and results in child’s participation. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Time-out does not cause distraction of others. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 




Hold Firmly 














Adult speaks firmly and calmly, without emotion. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Child is help only when necessary to prevent child 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


from hurting self or others. 
















Child is assisted in participating by adult holding. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Adult provides minimal expectations for participation. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Holding ends on a positive note. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Holding is brief and ends as soon as child indicates 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


ability to participate. 
















Remove from the Room 














Removal from the room is used when needed. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 



(Child is removed because he/she is out of 

control and may harm self or others; or 
- topic is so private that it cannot be 
discussed in from on the group. 

Circle PP if the adult removes the child for 
appropriate reasons.) 



The child understands why he/she was removed. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


The interpersonal exchange between the child and 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


the adult is positive. 

Someone is with the child when removed. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Child returns to group and participation. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 


Time away is as brief as possible. 


PP 


NP 


PN 


NN 


NO 


UN 


IN 
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Adult code number:. 



Pre Date: Activity: Number students: 

Post Date: Activity: Number students* 

RATING ADULT STRATEGIES: MIDDLE/HIGH SCHOOL TEACHERS 



Directions: This list of basic adult strategies includes descriptors proven effective in increasing student participation in planned 
educational activities. As you observe, keep a tally of the number of occurances of each strategy during the specified time period. 
Then, review the descriptors within each strategy category and rate the quality of use of each item. Keep in mind that “effectiveness” 
is defined as positive participation by the students in the activity. 

Circle PP if an item was used in a positive way by the teacher and effectively improved student participation. 

Circle PN if the strategy was used in a positive way by the teacher, but did NOT effectively improve student participation. 

Circle NP if the strategy was used in a negative way by the teacher, but effectively improved student participation. 

Circle NN if the strategy was used in a negative way by the teacher and did NOT effectively improved student participation. 

Circle NO if the strategy was not observed. 

Circle UN if the strategy was observed but unnecessary to improve student participation. 

Circle IN if the strategy was used inconsistently. 



Ignore Students 5 Behavior 



Adult appropriately ignores (does not interrupt) a student 
















who is involved in the task. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult appropriately ignores a student's behavior when the 

student is merely "testing" adult to elicit a reaction. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult does not ignore a student who is disturbing others. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult does not ignore a student when the student 
is in crisis or in need of adult assistance. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Use Motivating Curriculum/Content (Tasks) 



Tasks have high interest and cultural relevance. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Tasks elicit interest and participation of students. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Tasks are used to stimulate individual skills for success. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Tasks emphasize group processes. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Tasks appear to have real-life relevance for students. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Students are encouraged to talk about ideas, feelings, values. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Encourage and Praise 



Praise and encouragement are used with group & individuals. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Feedback focuses on positive elements of student’s participation. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult encourages positive peer feedback. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Praise and recognition of individual/group are authentic in quality. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Explain Procedures & Expectations (Structure) 



Before task begins, adult talks through/demonstrates. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Students understand the task & behavior expected. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult describes procedures to students. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Students are expected to successfully handle assignment independently. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Materials are organized for independent success with the task. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Organization/structure contribute to positive learning environment. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 
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Circle PP if an item was used in a positive way by the teacher and effectively improved student participation. 

Circle PN if the strategy was used in a positive way by the teacher, but did NOT effectively improve student participation. 
Circle NP if the strategy was used in a negative way by the teacher, but effectively improved student participation. 

Circle NN if the strategy was used in a negative way by the teacher and did NOT effectively improved student participation. 
Circle NO if the strategy was not observed. 

Circle UN if the strategy was observed but unnecessary to improve student participation. 

Circle IN if the strategy was used inconsistently. 



Model Positive Relationships 



Adult is positive model for expected behavior. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult conveys fairness toward each student. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult listens and responds with consideration/respect. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult-student relationships depict genuine respect. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Individual & group interactions are constructive in quality. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult uses positive humor to neutralize a situation. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult clarifies situation rather than giving commands. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Signal/Convey Awareness 



Adult moves near student as needed. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult uses a. glance or comment to signal awareness of actions. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult shows interest and support without interrupting task. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Redirect/Refocus Student to Task 


Adult uses redirection, statements, or questions 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


to refocus student on the task. 
















Students with lagging attention are re-motivated. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult simplifies procedure or reviews steps to clarify. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Reflect Confidence in Student (when student is in difficulty) 



Small accomplishments are recognized. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult reflects on the positive qualities in student. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult offers supportive comments to convey confidence in student. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Ask Student/Group to Review Rules 



Rules are concise and realistic. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Rules are built on positive values. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Rules focus on positive outcomes. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Student /group are asked to review rules. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Classroom rules focus on helping students be successful. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Rules are meaningful to students 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Extramural rules are discussed when appropriate. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Consequences for breaking rules are stated constructively. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Students are encouraged to modify rules as needed. 


PP 


■ PN 


NP 


NN 


NO 


UN 


IN 
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Circle PP if an item was used in a positive way by the teacher and effectively improved student participation. 

Circle PN if the strategy was used in a positive way by the teacher, but did NOT effectively improve student participation. 
Circle NP if the strategy was used in a negative way by the teacher, but effectively improved student participation. 

Circle NN if the strategy was used in a negative way by the teacher and did NOT effectively improved student participation. 
Circle NO if the strategy was not observed. 

Circle UN if the strategy was observed but unnecessary to improve student participation. 

Circle IN if the strategy was used inconsistently. 



Connect Students’ Words and Actions to Feelings (Interpretation) 



Interpretation is used when appropriate and does not 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


require a response from the student. 

Adult recognizes feelings behind student’s behavior. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult acknowledges student’s issue. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult is not judgmental. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult conveys understanding of student’s feelings. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult puts issue in reality context (e.g., a natural CQnsequence) 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Suggest Voluntary Time-out in Room 



Time-out from the group is suggested if needed. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Student understands why he/she should take time-out. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


The interpersonal exchange between student and adult is positive. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Time-out is brief and results in student’s participation. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



Confront Unacceptable Behavior 



Adult speaks firmly and without emotion. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Adult conveys specific expectation for alternative behavior. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Confrontation is used only when necessary 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


to control disruptive behavior. 
















Adult ignores student’s anger/hostility and uses 


PP 


PN 


NP . 


NN 


NO 


UN 


IN 


alternative strategies to stop unacceptable behavior. 
















Confrontation ends on positive note with student participation. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Remove from the Room 












Removal from the room is used when needed. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


Student is removed because he/she is out of 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


control and may harm self or others, or 
topic is so private that student cannot 
discuss it in front of the group. 
















The student understands why he/she was sent 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


from the room. 
















Student knows acceptable alternative behavior. 


PP 


PN 


NP 


NN 


NO 


UN 


IN 


The interpersonal exchange between the student and 


PP 


PN 


NP 


NN 


NO 


UN 


IN 



the adult is positive and ends on a positive note. 
(Student returns to group and participates.) 
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Interact with the PEGS! kids 

... and get them to participate in acceptable ways!' 



• Select preschool, elementary, or high school 

• Choose among various learning activities 

• View simulati ons of problem behaviors 

• Analyze what' s needed 

• Experiment with 12 practical intervention strategies 

• See animated reactions of students to your choices 

• Receive feedback 

• Increase your skills and decrease problem behaviors 





Information an d. flow to orders www.dowrdoadleariniitanigoCoifiHi 
Imservice tr ainin g aunud researclhi reports; www.tBgaLedtiL/dtttp 
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